Future of Coding - Jennifer Jacobs: Para & Dynamic Brushes

Episode Date: June 14, 2020

"Metaphors are important here." There's a small handful of people that I've been requested again and again to interview on the Future of Coding podcast. Jennifer Jacobs is one of those people. Her wor...k on Dynamic Brushes in particular, and parametric drawing in general, occupies a major intersection between disciplines and provides insights that we can all apply to our own work. This interview touches on childhood education, programming tools for both non-programmers and expert programmers, tangible interfaces, wearable and embodied computation, aesthetics, the relationship between academia and industry, means of evaluating the efficacy of projects, geometric encodings of first-order logic, symbolic representations, whether Scratch could exist outside MIT, and more. Jennifer does a wonderful job articulating the nature her own work, but also the works of her collaborators, peers, and influences, so that we come away with a great understanding for the broader spaces in which her research fits. Jennifer is already am important figure in our Future of Coding field, and I am very excited to follow her career and see all the places the impacts of her work will be felt. You'll notice right away that Steve is sitting in the interviewer chair this time. This is the first of a handful of episodes that Steve recorded in 2019 but didn't release. I'm planning to edit and release them throughout 2020, so you'll hear a bit more of Steve yet. The transcript for this episode was sponsored by Repl.it. Show notes and the full transcript are available here: https://futureofcoding.org/episodes/48Support us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.

Transcript
Discussion (0)
Starting point is 00:00:01 Just writing down what I think might be a good title for the podcast, designing tools for creative expression. Yeah, that's a reasonable title, yeah. Welcome to the Future of Coding. I'm Ivan Rees. There's a small handful of people that I've been requested again and again to interview on Future of Coding podcast. Jennifer Jacobs is one of those people. Her work on dynamic brushes in particular and parametric drawing in general occupies a major intersection between disciplines and provides insights that we can all apply to our own work. This interview touches on childhood education,
Starting point is 00:00:45 programming tools for both non-programmers and expert programmers, tangible interfaces, wearable and embodied computation, aesthetics, the relationship between academia and industry, means of evaluating the efficacy of projects, geometric encodings of first-order logic, symbolic representations, whether Scratch could exist outside MIT, and more. Jennifer does a wonderful job articulating the nature of her own work, but also the works of her collaborators, peers, and influences, so that we come away with a great understanding for the broader spaces in which her research fits. Jennifer is already an important figure in our future of coding field, and I am very excited to follow her career and see all
Starting point is 00:01:31 the places the impacts of her work will be felt. You'll notice right away that Steve is sitting in the interviewer chair this time. This is the first of a handful of episodes that he recorded in 2019 but didn't release. I'm planning to edit and release them throughout 2020, so you'll hear a bit more of Steve yet. The show notes for this episode are full of links to all the various projects and collaborators and inspirations that Jennifer references in the interview, along with a video of her demoing Para at the Adobe Max 2014 conference. You can find those show notes at futureofcoding.org slash episodes slash 48. There's also a full transcript if you'd prefer to read this interview rather than listen to it.
Starting point is 00:02:20 One final note before the interview starts. As the host of this podcast and the steward of our community, I want to say something in my official capacity. Black lives matter. The consequences of racism in the U.S., here where I am in Canada, and around the world permeate every facet of life and society. It will take a constant effort over many lifetimes to achieve justice, and all of us can play a role in making that happen. There are many ways we technologists can be deliberate in our own work to be sure we are empowering the people who need it most. Right now, I'm just going to look at one small piece, our community. Over the past six months, we've had a number of deep probing discussions about the need for greater diversity in our membership, in the guests that appear on this podcast, and in computing broadly.
Starting point is 00:03:12 One outcome of those discussions has been our code of conduct, which I'd like to read a few sentences from. One of many principles held by the community is that computing currently reflects the interests of a narrow minority of people. A directed effort is needed to broaden the accessibility of computing and amplify the influence of historically underrepresented people in shaping its future. The tools for thought we want to build aren't just to help us do more thinking about our tools. We're trying to make tools to help people solve real problems in the world. That means we need to be able to talk about these problems. We also need to be able to recognize and discuss problems within our own cultures, like the lack of diversity.
Starting point is 00:03:57 Those are just words. They are necessary, but not sufficient. I've received a number of suggestions for concrete actions we can take as a community, and that I can take as the community's steward, to make this a more welcoming and inclusive place. I'm doing my best to put them all into effect, and am excited to share more of that work with you when it is ready. Like accessibility, a focus on justice benefits not just the people who are underrepresented, but everyone. So if you haven't yet reflected on how to make your own work embody the values of justice and inclusion, or moving power from those who have too much to those who don't, I implore
Starting point is 00:04:35 you to do that. Finally, if there are things you would like to see me do, or changes you'd like to see in our community to make it a more empowering space for black and brown people in particular, I would be grateful to hear from you. Welcome, Jennifer. Thank you. Happy to be here. Yeah, I'm very excited for this conversation. And in particular, I think a few of my longtime listeners are excited too, because you have the honor of being requested.
Starting point is 00:05:11 Not all of my guests were requested, but you were. Oh, that's always nice. So I thought maybe we'd start with your background. I know you've worked at a number of research labs that would be familiar to people in our community. You've worked at the Media Lab, and it sounds like in two different groups at the Media Lab research labs that would be familiar to people in our community. You've worked at the Media Lab, and it sounds like in two different groups at the Media Lab. Is that right? Yes, Hilo Tech and Lifelong Kindergarten. So yeah, I think most of us are pretty familiar with the Lifelong Kindergarten
Starting point is 00:05:35 because it's Mitch Resnick's group, and it's the group that made Scratch. But I haven't heard of the Hilo Tech group. What's that? Yeah, Hilo Tech is a group that was founded by leah beakley um and leah is probably best known for uh being the creator of the lily pad arduino oh yeah which is a version of the original arduino but fundamentally redesigned to support e-tech style based electronics as opposed to more traditional electronics. So rather than breadboarding, you connect to the lily pad with conductive thread and there's a whole set of components that go with it that also are connected by sewing and conductive thread. The lily pad is
Starting point is 00:06:17 kind of something I knew about before I was interested in the media lab, but really a seminal example of how redesigning an existing piece of technology enables not just a different type of application for that technology, but also really broadens participation to a totally different group of people. And in this case, it was people who enjoy sewing or working with textiles. So Hilo Tech was a group that was really founded around this notion of integrating emerging or highly technical platforms like electronics, computational design and coding with established or traditional or sometimes what are considered like low forms of making in terms of arts and crafts, fine art, traditional craftsmanship, but also arts and crafts in the sense that people think of that as more of an approachable or hobbyist or accessible form of making. So the group was all structured around different ways of supporting both communities and different types of
Starting point is 00:07:17 practices in combining these different modes of making and different materials. So there was a lot of work in the group being done around combining electronics and paper craft, and this is being done by Jiqi, and she now has a company called Chibitronics that produces these electronic stickers or paper craft and electronics kits. Also Dave Mellis, who was one of the original Arduino team, did a lot of work in the group to help people make their own consumer electronics. So radios, computer mice, speakers, and then his most complex electronic product that you could build yourself is a DIY cell phone. There was also a lot of work that extended the e-textiles platform or the lily pad platforms emily level in the group did a lot of work around supporting community engagement and kind of creating a community
Starting point is 00:08:12 platform and sharing platform for the lily pad called lily pond and she also did some work in building different versions of the lily pad that were simpler and easier to get started with hannah perner wilson was also in the group. She's a rock star. All these people are rock stars, but she really took the integration of craft and electronics really far by combining her thesis was this work called A Kid of No Parts, which was basically looking at all the ways you could repurpose materials that were not traditionally thought of as electronic materials into platforms for building electronics and building sensors. And so she did a lot of knitted resistive sensors that could be then integrated into
Starting point is 00:08:53 clothing. She did work with conductive paints to create painted speakers on found materials. She's also one of the people I respect the most in terms of dissemination and documentation, where everything she did is very carefully documented and presented online. And if you're all interested in craft-based electronics, Hannah is really the master in that area, in addition to the work that Leah did. But Leah really brought this amazing group of people together. And then I was lucky enough to be able to join it for my master's. And that was really where I first got excited about this idea of combining code as a design tool and then as a design tool for making physical objects by integrating it with digital fabrication.
Starting point is 00:09:36 And so Leah was really the person who inspired me to start looking at the ways in which we can use computer programming to build really complex forms and objects and then turn those things into real things by using 3D printing or laser cutting or computer-controlled milling. So yeah, it's a quick overview of what Hyla Tech does, but it's no longer a research group at the Media Lab, but we still maintain an active community independently, and we actually have meetups biannually, and we're actually having one this summer in Kentucky. So really excited about that. Oh, wow. And it's just for people who are part of the group, or is it open to the public? No, it's still kind of an internal element. So the original members of the group meet up and share our research and share our work.
Starting point is 00:10:22 We've kind of all spread out across the world at this point. I don't know what this says about the type of work that I'm interested in doing, but often it's something that is at odds with a lot of like institutions. And so the media lab itself was, I think there's a lot of really amazing things about the media lab. There were aspects of high-low tech that weren't always understood or supported there. And I think there's a lot of really amazing things about the Media Lab. There were aspects of Hylotech that weren't always understood or supported there. And I think, you know, the fact that Hylotech is not a research group there anymore is something that's acknowledged as a loss. But it's also difficult to have non-traditional engineering communities in what are often pretty traditional or rigid engineering communities. And I think high-low tech was an example of that. So it still exists in terms of the work that we are doing as individuals
Starting point is 00:11:13 and Leah is doing, but is no longer a research group at MIT, unfortunately. So I thought the name, I didn't quite get it until you were just explaining it now, high-low tech. There's this saying or this idea that high tech or technology is just things that didn't exist when you were born. I think today's definition of high tech is things that didn't exist when you were born that were likely made in California. That's, I think, an approximate definition for what high tech is. I think what's kind of interesting about the high-low tech group
Starting point is 00:11:43 is that it's kind of like trying to take things that were made after you were born, but make them as usable things that you grew up with. Yeah, a bunch of it is thinking about ways in which we can make different approaches to making in different media and material for making more approachable or more inviting or more relevant to different groups of people. And often that's groups of people that traditionally have been excluded or viewed as not relevant to them. And so in some ways that's lowering barriers to entry and making things easier. Although, as I know, you're very aware of and your audience is very aware of what it means to make something easier is not a simple notion. There's a lot to unpack there. But it's also often just this notion of, I would say, what are the
Starting point is 00:12:32 expressive opportunities that come with reframing a form of making along the traditions of another form of making? So in traditional circuit design, the trace is primarily viewed through a notion and like the trace on the circuit board. Be careful. Trace can mean something different in programming. The actual circuits themselves are primarily viewed purely from a functional and efficiency perspective. So is it thick enough to support the necessary electrical properties and does it connect the necessary components? But if you translate that to the trace is also something that you're embroidering with conductive thread or you're drawing with conductive ink then it's both a means to transmit electricity and have a functioning
Starting point is 00:13:17 circuit but it also becomes a decorative or an aesthetic element and that's something that comes out of traditions of drawing and embroidery and sewing, where functional aspects are also things that contribute to the visual properties, the beauty of a piece. And that's something that I think everyone in high-low tech felt was very powerful and very meaningful. So it's an opportunity to engage people who maybe have an interest in drawing or have an interest in sewing or in craft in seeing themselves as producers of technology, which is very important. But it's also an opportunity to re-envision what technology might look like or what these materials might mean and how they might be used to people who kind of have a closed notion of what a circuit is or what applications computer programming is used for. And so one of the really interesting experiences I had in the group is when I was first doing computational design workshops with processing and I was having people write code to make designs for lamps, for example. I would work with people who had never
Starting point is 00:14:21 programmed before and this was their first introduction to programming and they would be interested in that. But I'd also work with people who had never programmed before, and this was their first introduction to programming, and they would be interested in that. But I'd also work with experienced programmers and software developers who would say, oh, I had no idea that code could be used to make beautiful forms. I feel like both of those are powerful outcomes. So I find myself wondering how you identify, and I have these two myths in my head that I think could apply. One is the myth of the guy who brought fire from the gods. Oh, Prometheus. Prometheus, yes.
Starting point is 00:14:49 So you could see yourself as Prometheus as being like a technologist who's bringing technology to artists. Or you could see yourself as more like Jack in Jack and the Beanstalk, like someone who goes up to the gods and steals their stuff and then comes back down. So do you identify more as a technologist who's democratizing technology or as an artist who's trying to claim technology for themselves? Yeah. I would say there are elements of both of those that are things that I think are important and interesting and supporting,
Starting point is 00:15:20 but I wouldn't say I view myself precisely through either of those lenses. So one thing that I continually struggle with is the language that I use to describe what I'm doing and who I'm doing it for. And a lot of this comes up in talking about artists versus programmers. The reality is that while not all artists program, it's also not an accurate distinction to say that there are programmers and then there are artists. Much of the work I've done has been inspired by artists who are some of the best programmers that I also have ever met.
Starting point is 00:15:56 And also some of the best software developers and application developers that I've ever met. So I would see it more as trying to explain to artists why programming is so great or help make it easier for artists to program because artists do very difficult things all the time and learn very difficult things all the time. But more thinking from the perspective of what are domains of art that could really benefit from some of the powerful affordances of programming? Because I believe that, frankly, at this point, everything can benefit from programming if it's presented in the right context,
Starting point is 00:16:29 but that we might rethink how we develop programming environments or programming languages to support these specific domains. And that's less about, I would say, just making it easier for some artists to use programming and more about thinking a little bit about changing the idea of what programming might be or changing the notion of how it might connect
Starting point is 00:16:54 to different forms of expression. And at the end of the day, this comes down to my own experience and the things that motivated me personally. I often talk informally about my work as being a series of productive failures and trying to mash symbolic programming and drawing together. And because there are these two forms of creation that I feel are incredibly powerful personally, but have fundamentally different properties
Starting point is 00:17:24 in many respects. And I can talk more about what those are at length. But a lot of the work that I've done has been trying to approach different ways of how can we get these two things to be integrated more closely together? How can we make the experience of programming more like drawing? Or how can we make the process of programming reflect some of the key forms of expression of drawing and that's translated out into a bunch of different domains beyond drawing I think it's more of a notion of taking these two forms of expression that are really powerful and thinking about different representations different interfaces even different forms of teaching and
Starting point is 00:18:02 facilitation that allow us to combine them as opposed to introducing this audience like artists who are previously unaware of this medium programming to it and helping them to get started that's definitely a part of it but not the whole story from the other perspective I think you use the metaphor of jack and the Beanstalk. I think there's another angle of this work or a question that I ask myself, which is what if at all value does my work provide for more established programming communities like software development? And that's only been a question that I've been focusing on relatively recently. And I think it's not the primary focus of my work, but there are a lot of things that I've been looking at in terms of how artists program or how artists work, and this is generalizing to some extent, but often true,
Starting point is 00:18:54 but they often engage in an exploratory working practice where they don't start with a clearly defined problem and say, okay, now I just need to break this into smaller problems and solve it. They have a process where they're open to mistakes, they're open to variation, and they kind of explore as they make. Victor has talked about this in a lot of his writing as well. That's something that's true for artists and often goes against very traditional notions of what good programming practice should be or what good software development should be. But I also think that software developers work in an exploratory manner. And other people have written about this in many cases. And I think that it's impossible to predict and plan every aspect of a programming project before you execute on it.
Starting point is 00:19:41 There are always surprises and unexpected things and experimentation. So I think there are things we can learn from how more messy forms of creation or more messy creative practitioners use programming that might help us make better programming tools for people like software developers or other communities. But I haven't explored that direction in my work as much. Just to clarify, when I was talking about the Jack and the Beanstalk metaphor, the metaphor was artists are more normal people and then the giants are the programmers. And so you're like stealing some of the programmer magic and bring it down to the artists. Yeah. Yeah. I'm sorry. I kind of... You flipped the metaphor as like artists where the people... Because you could do it either way. Yeah. Yeah.
Starting point is 00:20:25 I mean, there is so much magic in programming, but I don't. I guess just to give you examples of people that come to mind, like Seymour Papert to me seems like someone who was like really a mathematician who was trying to like bring mathematics to people who didn't like mathematics. As opposed to some like an artist who wanted to use mathematics in their work. He was very much, I felt, like a democratizer when there are other people who are like scrappy artists who just take whatever they can and embed it in their work. I mean, I do think there's this problematic notion, and I've heard this addressed on your podcast before,
Starting point is 00:21:03 but this problematic notion of certain types of people being equipped to be good programmers and other types of people being less equipped or a notion of exceptionalism in certain forms of programming. That's unfortunate and not surprising, but I do like to try to break down that idea a little bit that while programming is an incredibly powerful form of expression, while abstraction has value in so many different forms of design and so many different forms of expression, while symbolic mathematics has value in so many different forms of expression, programming is too vast to say that there is one personality type or one intelligence type that really aligns with it. And if you don't have that, you're never going to truly
Starting point is 00:21:52 be effective in it. That's definitely a belief I had about myself getting started in programming, and it was totally by accident that I got started programming. But I see it now with some of the people that I work with who, for example, a lot of artists who use visual programming languages like Grasshopper, which is the node-based data flow language for Rhino, which is a very powerful programming language. And I'll have people tell me, oh, I don't know how to program. I just use Grasshopper. It's like, why is that not programming? And where did you get that idea? Because it is a visual programming language you know and there are people but i think fewer and fewer who don't consider visual
Starting point is 00:22:30 programming programming but um definitely not uh from where i'm sitting fewer and fewer right right right okay well i've had the good fortune of being in a very supportive environment in lifelong kindergarten around this i guess and i'm still learning a bit about how to rest the world. Yeah. You're like in Mecca. You're like, oh, everyone believes in the same God as me. Everything's wonderful. But so I do think it's important to dispel those ideas and dispel notions of also like where programming is relevant. I don't know how much oxygen I even want to give to that space, but needless to say, I focus more on not so much,
Starting point is 00:23:10 you know, is this programming or is this not programming? And more, what are different people trying to make? How can the ideas or the approaches embedded in programming support them? What are the barriers? And how can we help more people make things by building different interfaces or environments or tools that get around some of the barriers? And how can we help more people make things by building different interfaces or environments or tools
Starting point is 00:23:26 that get around some of those barriers? And there's so many questions and challenges in there that at the end of the day, there's somebody who looks at what I do and say, yeah, but you don't have conditionals in this language. So it's not really a programming language. I'm like, well, there will be other issues where we need conditionals,
Starting point is 00:23:45 but I want that to be driven by the people I'm trying to support or the work that I'm trying to support, not by the fact that there's a notion that to be real programming, you need conditionals. I feel like I want to just keep talking about this sort of stuff, but I think it probably might help our audience to talk more concretely about some work you've done. Yeah, absolutely.
Starting point is 00:24:06 Actually, before we do that, I want to just ask you a question, which I might end up cutting, but I'm curious to know the answer. Sure. I really enjoyed the who's who of Hilo Tech and all the interesting work happening there. And I was wondering if you would do the same for the Lifelong Kindergarten group. I can tell you who was present when I was in Lifelong Kindergarten because it's since grown. So the group is led by Mitch Resnick, who I am sure many people are familiar with, but he was a student of Seymour Papert and one of the main designers and really the continued architect of the Scratch programming language. Although Scratch is really a community effort, It's come together
Starting point is 00:24:46 from the contributions of a lot of different people. While I was there, there were a few other grad students that really influenced me. So, Rikaros Roke, who is now faculty at CU Boulder in the information science department, I think. She was doing a lot of work around facilitation in computational learning, specifically related to families. So how do you help not just kids learn to program, but how do you engage people in computational learning in the context of a family?
Starting point is 00:25:20 And the cool thing about that work is that she is an immigrant, comes from an immigrant family, was focusing on working with other immigrant families by and large as a means to think about what are some of the constraints that different communities people have when engaging in different types of learning. And how can we have this be a process not just to help people develop new skills, but also something that engages parents and their children together in working with a new technology. Rick Rose's work really stands out to me because facilitation is such an important aspect of how we think about people learning programming.
Starting point is 00:25:56 Often the work in my domain that gets a lot of focus in research is what can an environment do? What can a technology do to make learning easier? But there's also so many important questions about what it looks like to facilitate a learning environment, what the interactions look like between the people who are quote-unquote experts and the people who are non-experts. And Rick Rose is one of the people who's really doing deep work in that space. Simon Dudescupta was another grad student who was in the group when I was there. He is doing really interesting work of combining data science and programming for young people,
Starting point is 00:26:39 just like Seymour Papert and Mitch said that kids are programmers and we should build programming languages for them. Saimindu has kind of extended that to say that kids can be data scientists. And he's looked at ways to use Scratch as a way to help kids do programming with data around their own data. So the Scratch community has a lot of opportunities in that space where kids can get data about how many, you know, the way people have viewed their projects, the people they've worked with in the community. And so he's done a lot of really interesting research in that area. And I think that there are some scratch extensions
Starting point is 00:27:07 that have come out of it that specifically are geared towards programming with external data. So, oh man, there were a lot of other people in the group when I was there. Eric Rosenbaum was there. We overlapped. He and Jay Silver did the Makey Makey.
Starting point is 00:27:26 And Eric has done a lot of really great work around computational music creation that's continuing to be embedded in Scratch. One of the great things is that a lot of students in the group are also members of the Scratch team and will come back in various ways and continue to contribute to Scratch. I think Eric is actually now working as a member of the Scratch team. Natalie Rusk was there as a research scientist. I believe she's still there. Yeah, I've heard her name before. Yeah, Natalie is another big component of Scratch and also has done really thoughtful work in the space of motivation and understanding what drives people to engage in difficult learning tasks or learning challenges. Then there's the whole Scratch team, which one of the things I feel
Starting point is 00:28:13 really fortunate about being in lifelong kindergarten was not just interacting with the grad students, but the fact that there's this basically small company of people who are maintaining and designing and doing the engineering for the Scratch community and Scratch programming environment. And they're such a wonderful group of people, but also have such a wealth of knowledge around design, around working with people, around teaching. And a lot of the work that I've done was directly influenced by talking with those people about how they thought about designing visual programming languages or how they thought about developing documentation. And I think it's something that is maybe often not present in academia where you're maybe working in a simplified
Starting point is 00:28:55 or more less ecologically valid context and less aware of what people in the quote-unquote real world are doing. There's too many people in lifelong kindergarten to talk about, but those are a few people that really influenced me. I think for people who aren't in academia, academia sometimes seems overwhelmingly large. But when you meet a few people who are in academia and you just like ask like, who are your friends? Like, who are the people that you've worked with and are interesting? And they just like list 10 people and you're like, oh, I can learn about those 10 people. Yeah. Yeah. There's too many people to talk about. But the other nice thing about both Hilotech and Lifelong Kindergarten is everyone I've encountered in those groups has been a very kind person as well. And so if people are interested and they look into, you know, Simon do a Rick Rose's work, I encourage them to reach
Starting point is 00:29:39 out because they're all very kind and interested in sharing and connecting with people. Cool. Okay, so now let's get into some of that more concrete stuff of the work you've done. There was one talk of yours I saw, you had two different projects. One project, you did some testing and evaluation, then you did a second project. So you're probably thinking of Para and Dynamic Brushes, I'm guessing? Yes. Got it. I forgot the name Para. The second one and Dynamic Brushes, I'm guessing? Got it. I forgot the name Para. The second one, Dynamic Brushes, I'm familiar with. Yeah, it's not a great name, but we needed a name.
Starting point is 00:30:15 Well, I like Dynamic Brushes a lot, actually. I have trouble with the phrase procedural art because procedures are just one way of describing things. Yeah, trouble with it too but i haven't found a better word so i guess like i like maybe dynamic art or computational art you know something more abstract than procedural because you know yeah yeah um i i'm continuing to refine the language I use. I chose procedural because it's one that both in the art community is used and also historically with regards to art made with computers and also to some extent art that is not made with a digital computer, but arguably is computational, like instructional art for example um solo wit
Starting point is 00:31:06 is kind of the canonical example in that space that's interesting that there's this connection to instructional art even before computers or like giving like a recipe of instructions even before computers were around yeah well we can go way back into the history of all kind like all kinds of forms of algorithmic production that existed before computers yeah interesting yeah yeah like you could yeah i can imagine yeah like uh an architect could like you know explain to the brick layers where they want the bricks to go yeah and i'm sure you've are familiar with christopher alexander's book a pattern language and all the work he's done that's all about arguably abstraction in in, in architecture, in structure. But knitting, weaving are kind of the canonical
Starting point is 00:31:51 examples of algorithmic forms of making. It's interesting because, not to go on too much of a tangent, but it's hard not to. A lot of artists think about what they do algorithmically, even if they don't program, because artists are highly reflective in their process. They're very interested in what they make, but they're also very interested in how they make it. So for example, Chantel Martin, who's an artist that I'm very inspired by and had a lot of conversations with her about her process and drawing. She creates these giant like large-scale drawings all improvisationally and all seemingly without a plan but she talks about the fact that oh yeah I understand this general structure of when I make a decision to vary the line, how long that I draw, the general distance
Starting point is 00:32:42 between different elements of my drawing, you know, when I choose to create these different singularities or inner drawing, like exceptions, and she doesn't program, or she hadn't really when we were first talking, but she was thinking about her work very procedurally, and she was then very drawn to the notion of programming, and since gone in to do work in this space, the idea not just that it would let her create different things but that oh this might help me see the process that I use through a different lens through a different representation is she one of the artists that you commissioned to use your tools to try it out not yet hopefully soon she's she's uh unsurprisingly really blown up recently.
Starting point is 00:33:26 Not recently, but yeah, this is the problem of talking with really talented people is they're very successful and then they have a lot of projects to do. We've talked about doing projects together and hopefully eventually that will happen, but she's also doing a lot of work on her own, right? Yeah, she's very successful at this point in time,
Starting point is 00:33:46 which is great. So I liked in your presentation about these two projects how you framed it that you were doing interplay of the dictionary systems engineering. There was this cool graphic that you started from certain motivations. Anyways, so maybe you could walk us through the process. It builds very much on the notion of user-centered design or other human-centered forms of design practice, where it is to start with an understanding of an existing domain and examining what people are doing in that domain.
Starting point is 00:34:23 What are the opportunities and what are the challenges, and then starting to try and build tools and technologies that try to address some of those challenges. But the idea being that you're really starting first by trying to understand the practices of the people you're trying to support. And so in the case of artists, like I do a lot of work initially with observing artists at work in their studios, interviewing them, analyzing their work and kind of making hypotheses about how they might be doing certain things and then asking them about this and talking with them about it. And then trying to prototype tools that might align with some of the things they're trying to do. To start with Para, one of the things that I noticed really early on when doing workshops with people with processing is that often people
Starting point is 00:35:08 were coming from a background of using direct manipulation tools like Illustrator, and processing generates vector graphics, so it's in many ways really powerful to combine it with a tool like Illustrator. But there was this big disconnect, both moving between these different interfaces, symbolic textual programming and direct manipulation of vector graphics, but also these very different representations where you're using this imperative representation and processing where explicitly every transformation you're describing, whereas in a direct manipulation tool, it doesn't matter how many times you move something. All that matters is what the final position of that object is, for example. And so kind of seeing
Starting point is 00:35:51 these tensions come out and thinking about and then informing them through conversations with people about why they choose to use an imperative programming tool like processing or why an artist might choose instead to use a tool like Illustrator. And that really led into some of the early prototypes for Para, which is this direct manipulation procedural art tool. And so I don't call it a programming tool because there's no symbolic code. There's no code that's displayed to the person using the system. But there are types of things that you can do with it, like describe constraints, have iterative variation between multiple objects
Starting point is 00:36:31 that normally you would use a programming language to achieve these same relationships. And the development of Para, going back to this interdisciplinary systems engineering idea, the challenge in building the system is, as you're building it, you're making all the, everyone who's worked in software development knows this, you're making all these individual decisions at any point
Starting point is 00:36:53 that could be really impactful or are just necessary to get the thing up and running. And the task is to understand at some point how to inform those decisions. And so the approach that we've taken is to actually do iterative testing of the system with professional artists using it. So as soon as the system is in any shape for anybody to use it, which is often like way too early for my comfort, but probably too late in
Starting point is 00:37:21 some ways, I should probably continue to try to do it earlier. We'll have professional artists use it and try to make work with it and observe them using it and actively modify it in response to some of the challenges that they observe. With Para, we had a professional painter use the system for two weeks while we looked at what she was doing. This is the painter Kim Smith, who's also at the Media Lab. And then revised elements of both the interface and also some aspects of the underlying representation in response to things that just didn't make any sense. So in some ways, the ways we were representing lists were really powerful from an abstract notion in that you create a list and then really it was very was very simple it was
Starting point is 00:38:05 just a an ordered collection of vector graphics but not very useful from an artistic standpoint if you create this collection and you have no immediate functionality that comes out of it so we we added the ability when you create a list that it is maybe getting too deep in the weeds but that it automatically had some constraints on those objects imposed so that when you would start to manipulate them through direct manipulation, you would get useful changes across the collection of the entire objects.
Starting point is 00:38:31 This is something that came directly out of working with this one artist and in the process of watching what she did with it. And that process is something that improves the tool itself and also helps us to understand more general notions of like, how does it actually apply to longer term use? Does it support the creation of different types of art as well as supporting kind of these interesting small interactions? At what point
Starting point is 00:38:58 does it stop becoming interesting for the artist? And how do we need to think about different forms of expression that we should enable? And then also, really, what are the limitations of the tool? So really, towards the end of the second week, Kim started using Para to make more illustrative work in the system as opposed to abstract forms. So she was using the ability to automatically manipulate large collections of objects, to create patterning in an illustration of an animal and some plants. That was really interesting to see and was really exciting, but we also noticed right away that compared to some of the illustrations she had done, you know, with watercolor by hand, the fact that she was using
Starting point is 00:39:37 vector graphics was a kind of a big constraint in terms of those forms of expression. And so that's where this like cycle kind of feeds back into building into the next project, where you look at what people have done, it helps you improve the existing system, but it also points to these directions to new work and new systems. And so without doing the studies with Para, with artists, we wouldn't have thought more about, well, should we examine these assumptions we've made around vector graphics and direct manipulation of vector graphics as a way of supporting more expressive forms of procedural art for artists who work by hand and focus more on this idea of drawing and actually
Starting point is 00:40:17 drawing with a pencil or drawing with a stylus. And that's where dynamic brushes came from. And that's not to say that I think that Para was a dead end. In fact, it's something that we're still working on. And that project was a collaboration with Joel Brandt and Radimir Mech and also Sumit Goja. But Joel and Radimir are both research scientists at Adobe. And, you know, there are ideas from Para that are continuing to percolate through through adobe in different forms but it also sort of opened up new new ideas or or different directions that we might go and and that's kind of where dynamic brushes came from yeah before we continue on in the story i wanted to just talk a bit about um para i think just just to give people a mental picture if something didn't come to mind uh i think the tool looks a bit like photoshop's like a normal graphics editor it
Starting point is 00:41:09 actually looks more like illustrator but yeah it's similar yeah it has a toolbar it has a what appears to be a layers panel and then a canvas and a few other things yeah yeah yeah that's that's yeah i just want to give people a bit of a mental image and then the way i I thought of it is, like you said, it's not a programming language. It's first and foremost a tool like that, but then it seems kind of augmented with these cool computational constraints and linkages, which I've seen actually a few tools start to incorporate. For example, Figma has this new components, like master and instance ones, that you can change the master and all the instances will change too so it seems like this tool was exploring that space yeah i should say um there's been work in the past that has some of these same elements and a major frustration i have in in
Starting point is 00:41:56 research in general is that i feel like we're often discouraged from building on prior strong work as opposed to um distinguishing our work as completely novel. But Para has new things in it, and I would say represents a different direction than some previous work. But everything going back from Sketchpad to Morphic to the work that Toby Shackman has done in Apparatus to the work that Rubiat Kazi has done with Kitty, and then also obviously all of Brett's work in dynamic data visualizations. It's definitely built on a strong foundation of prior works. I would say that the things that maybe don't get focused on in Para as much that are important to me are how we tried to position those things with respect to the specific domain of
Starting point is 00:42:46 visual art and illustration and then also really how we evaluated it with artists in terms of looking at the types of work that it could enable and the ability to support both more abstract forms of procedural art but also these more illustrative forms with animals and plants and so on. But yeah, it's built on this foundation and inspired by and hopefully contributes to this larger body of work that I think is really important. Oh, also sketch and sketch is an important example in that space too. But yeah, this body of work that's looking at what are the ways in which we can express procedural relationships, programmatic relationships through direct manipulation. And there's so much exciting work that's been done in that space and
Starting point is 00:43:30 so many difficult challenges to continue tackling. So the list improvement you made that you went over, I thought that sounded interesting. It sounded like before when someone would make a list, it would just be a collection of items, but they wouldn't be linked at all. And then so you changed it so when you put things in the list, they are automatically constrained to each other in some way? Yeah, I mean, one of the first ideas we had was really just what are powerful ideas from more conventional forms of programming that we can apply to the direct manipulation context and have them so represented. And the notion of a list generally is just something that can serve many applications, but in itself is specifying an order and a collection of data,
Starting point is 00:44:13 but not specifying operations on that data necessarily. And although that is useful to have sort of bare bones lists off the bat, as we quickly observed when you're coming from the space of direct manipulation and also maybe not thinking about collections of artwork in that way. What's really powerful about Para is that you can define a relationship and then start to play with it by manipulating the shapes themselves, by manipulating the actual artwork. And the powerful aspect of direct manipulation in general is that immediate feedback where you change something and immediately get a sense of how it changes everything else.
Starting point is 00:44:53 And so to have built-in constraints when you're applying lists, so that when I apply a list to 10 objects and then drag the last object and it redistributes the objects gives people immediately a sense of some of the potentials of using those types of relationships as opposed to they describe a list and really the structure is there but but nothing happens immediately there was a whole bunch of additional work in general we did in para of thinking about oh what are these really complex types of procedural relationships you might create. So we had something that looked a little bit like prototype inheritance in the system, where you would have something that when you duplicated an object, it would maintain the same properties as its parent until you explicitly manipulated one of those properties, and then it would have its own value for that. And that was
Starting point is 00:45:41 very powerful, but you really had to have not only a deep understanding of where you might want to use that, but also it really upped the burden of how much complexity you had to manage in the system and how we might even represent those relationships or represent that complexity. And so in the end, we settled on this simple set of procedural concepts, constraints, lists, and declarative duplication because it kind of hit this sweet spot of letting you do things you couldn't do in existing direct manipulation tools, but not increasing the complexity so much that we really had to, you know, like it's very rare that you would encounter a situation where you're creating a cyclic constraint in para for example you can and the system will alert you if you're doing that but because you could do so much just with simple object to object constraints or do a single
Starting point is 00:46:34 constraint and map it from one list to another list a lot of that complexity was unnecessary i would say going forward one of the biggest challenges we have in thinking about direct manipulation as a format for expressing programmatic-like or procedural relationships is how do we help people debug these things? if you can't unpack them, if you can't evaluate them when they're functioning incorrectly, then it's not going to be any easier than other forms of programming. And to be fair, this is already an issue we have with regular direct manipulation systems where they don't necessarily always perform super well when you have really complicated compositions or large collections of artwork. Like the layers panel is not the greatest panel for manipulating an artwork with assets on the order of the thousands or even the hundreds. And there's people doing work to address this. But I would say like if and when I continue to do work in the space of direct manipulation
Starting point is 00:47:40 based procedural or computational art tools, debugging and understanding structure is really an area of interest for me. So one of the questions that I still have about this list improvement you made, did you simplify lists from this generic structure to have default behavior, to have standard behavior, or was it just that you gave lists a certain default that could be undone if they wanted it to be? No, we just gave them a default. You can always remove in Para. You can remove any constraint that's defined.
Starting point is 00:48:14 We didn't actually take away any functionality. We more chose what, for the context of this particular study, was an effective default. I remember, yeah, learnable programming does a great job of explaining why good defaults are just like so important, but there's such a simple way to improve the learnability of a tool. Yeah. I would say this is something that often gets not as much, I'm glad it was highlighted in that essay. I don't think it gets enough appreciation in general, like something that's often lost on people who don't understand the power of Scratch is not understanding some ideas of really powerful but simple starting points they provided people, how they introduce different concepts.
Starting point is 00:48:52 Microworlds is kind of a really important idea in this space. But then I would say that I think there are a few examples of places where Scratch goes kind of too far and they don't let you unpack the glide like the glide command i have a lot of trouble with because interesting and i i like you know tell my told my students you know to never use it to you know and it's harder initially because they have to learn uh they have to like point their object in the way they want it to go and then move in a loop so it's like three blocks instead of one but it's so much more powerful to understand the primitives in that
Starting point is 00:49:25 way than it is to glide is is so limited um and so yeah anyways yeah no it's that's interesting to hear i mean i think i think this is the general tension you get when you have a pretty diverse audience of people using a given tool. I know one of the challenges that the Scratch team is always dealing with is how can we support people who are doing really complex work in the system without making it so that when completely new people encounter it, they might be encountering types of code or types of blocks that make it really difficult for them to do simple things. And there's always a push and pull in terms of how you resolve those challenges. My interest as a researcher working in the space of designing creative systems is
Starting point is 00:50:17 finding good resolutions to some of those challenges, but moreover, trying to do a better job of documenting and explaining how those decisions were made and why they were made. I feel like this is a struggle in writing about any software-based project is there's a lot of design knowledge and information that gets kind of glossed over because the process of building a software tool is made of so many important small design decisions that really converge to make the tool what it is but it's very difficult to communicate those decisions in a short and understandable manner to an outside audience there are other things that happen in para that i haven't been able to unpack and talk
Starting point is 00:51:03 about um and we're definitely returning to the system to continue to work on it in other areas. But yeah, this is an ongoing challenge in the work that I do. Let's talk about dynamic brushes now. And maybe you could pick up the story where you left off how the evaluation and feedback you got from Para led to the conception of dynamic brushes. Yeah, so it was sort of a convergence of factors one was really getting to the limitations of vector graphics where they're very useful for some forms of expression but if you're someone who draws or paints and many of the people that i was working with fit that description. There is so much
Starting point is 00:51:46 expressive power you get just from the movement of different people's hands. Just any anybody draws differently than anybody else no matter their level of experience. If you look at people who have a lot of experience in a form of manual draftsmanship let's say like painting or drawing, they have spent years refining how they move their hand to draw. I spoke at length with my former undergrad advisor, Michael Salter, who's a illustrator and studio artist. And he talked about how the opportunity of drawing is that the fastest way for him to express an idea in his head was the length of his arm and that he really refined this ability to draw and express you know ideas that came into his head very quickly and so wait sorry uh what about the length of his arm oh it's sort of a i should find the exact
Starting point is 00:52:38 quote um but but that the the you know the fastest transmission of an idea from his head to reality was basically this mapping between his head to the extension of his arm. I'm misquoting him slightly there. So apologies, Michael. Yeah. We're like someone who's good at talking. It might be the shortest path is from their head to their tongue. Yeah. Yeah.
Starting point is 00:53:00 Maybe. Yeah. Yeah. Yeah, yeah, maybe. Yeah, yeah. It's this common notion that has come up, the speed of drawing being something that's both powerful in terms of getting ideas out there, but also the thing that affects what it looks like. So Chantel talks about how she tries to draw without thinking very quickly, because if she thinks about it too much, it'll change her style. And it's almost like it has elements of like meditation in it i think to some extent i think seymour pap would really like this kind of conversation because it's all somatic
Starting point is 00:53:31 somatic knowledge body knowledge yeah yeah uh yeah there's definitely uh this notion of yeah embodied or tacit knowledge i think uh that idea was really what led to dynamic brushes of thinking, okay, let's take the same approach we took with Para, but reframe it around this notion of drawing by hand. So you're not using vector graphics, but what you're doing is drawing with a pencil, or in this case, like a digital stylus. And how can we use that as a starting point? There's any number of directions that might have taken but then I was also having a lot of conversations with artists who regularly used programming in their work and I focused on talking with artists who also did these works that
Starting point is 00:54:18 had what to me seemed like very hand-drawn qualities to them. So Eric Natsky and Emily Gobiel and Theo Watson. There is a common pattern there where they were building basically their own software that let them draw, but then procedurally manipulated aspects of their drawings. They were drawing with a stylus, and then Eric wrote software where it would take the stylus input and manipulate the line. This is simplifying it, but manipulate the line by sampling from some other external data like a JPEG image. If you look at his work, some of his work from this time, it has these really sweeping forms, but then this really complex variation on the weight of the stroke and the color of the stroke that's coming from this functionality of software he wrote.
Starting point is 00:55:09 And the same thing with Emily and Theo had a really interesting process where Emily comes from both a background in programming, but also in illustration and puppet design. And Theo comes from a background that's more rooted in software development and computer science. And they have this wonderful collaborative process where Theo would kind of write these ad hoc scripts in Open Frameworks, which is a tool he was one of the co-founders of, where it would do something really simple but really playful, like take the line that Emily was drawing and then also take microphone input and modulate some aspects of the line geometry based on the amplitude of the mic input for example or take some form she had drawn and then automatically do a circle packing algorithm within it and fill it with with circles and Emily would use these and then generate ideas of oh what if we did instead? And Theo would script up another tool as a variation, and they kind of work back and forth
Starting point is 00:56:09 and build these ad hoc drawing tools. It's like, oh, that's amazing. That's such a powerful process. Can we think about the idea of a programming language for drawing as something that is actually a programming language for building your own drawing tools that take your drawing input and transform it in some way that the artist dictates or describes. And that's really where Dynamic Brushes, that was kind of the start of that project, is that idea. I just have to say, you're a true computer scientist. You're like, you're starting with drawings. You're like, oh, well, this is nice. But like, how do I make these drawings a little bit more dynamic? And you're like, okay, I'm going to make a tool for artists to make dynamic drawings. Okay, that's a good idea.
Starting point is 00:56:49 Oh, wait, you know what? I need to make a tool that artists can use to make their own tools. Yeah, the next project is going to be, you're going to make a generic programming language that will allow people to make tools that will allow other people. Yeah, it's just, you're never going to stop. Yeah, no, it's turtles all the way down. Yeah, I think it's a natural process when your goal is to think about
Starting point is 00:57:11 how can I help people be expressive with programming? That's basically what programming enables you to do is build your own, tools is not always the right term, but your own subroutines, forms of automation. Often artists I've talked to who use programming extensively describe it as programming and it lets you build a medium and so it's useful to take other aspects of programming like oh can we just help people automate this one thing that everyone needs to do that's's powerful and that's valuable. But I think the more I think about helping different people be expressive, you really want to give them the full access to the
Starting point is 00:57:51 true power of programming, which is constructing your own software, basically. I like the way you're doing it in almost like this DSL kind of way. Yeah. It reminds me of like the LMK Steps project where you have like a main interface that's easier to use and you have a specialized interface to just specialize the first interface and then maybe an interface to specialize the specializer. Yeah, there's all kinds of problems that come out of that and I'm happy to talk about those at length. But I think in general, I'm a big proponent of this notion
Starting point is 00:58:21 of domain-specific languages and dynamic brushes. Although it brought a lot of challenges, I think the idea of, okay, let's build a language around this notion of managing drawing data with a visual programming language and then letting people work with those tools in a more traditional direct manipulation drawing environment. So the data from your stylus is now going to be represented as absolute position over time, or its relative position over time, or the change in force over time. And once again, this was a collaboration with Joel and Radomir, and then also Mitch was hugely influential in this work.
Starting point is 00:59:01 We borrowed a lot from work in the space of computational music manipulation, where they're using very simple waveforms to modulate sound or generate sound. And we then just simply applied them to modulating the line that was being drawn, the position of it, the geometry of it, also the stylistic properties of it. So you can have a brush in dynamic brushes that follows the stylus X and Y position, but modulates the geometric scale relative to the origin along a square wave. So you get this geometric variation, but then the position is totally determined by how the different person is drawing and so it's this incredibly simple program from one viewpoint the program itself is very short it's not always
Starting point is 00:59:51 a measure of simplicity in programming but um but then you give it to a different artist and each person makes something different with it whereas in para although i liked para a lot and i still love it a lot a simple program because it was using the same geometric primitives, there was a much higher threshold for different people generating different things off the bat. And so that's probably the most powerful thing about dynamic brushes is that it just brings in this form of data that's highly expressive and you can do a lot with simple programs. You can also do very complex things. And so one of the key goals of the system was, let's not just see what happens when we treat drawing as an input and let people manipulate it, but let's enable people to
Starting point is 01:00:31 create tools they couldn't make with other, that aren't available in other drawing software, like Illustrator or Photoshop, where they can have things drawn that correspond with what they're drawing by hand, but they can also have things being drawn automatically or independent of what they're drawing by hand. And so I don't know how in detail you want me to go into the programming representation. I do want you to go into it because I think it's fascinating. I just want to draw a mental picture and I'll do a quick sketch and you can fill in some of the details. Yeah, yeah. So dynamic brushes, the pictures I saw was you have iPad and a computer screen.
Starting point is 01:01:07 On the computer screen, it looked a bit like scratch. You had blocks that you could connect. Then on the iPad, you were drawing. The thing you were coding on the desktop screen was the brush. As you changed the brush on the desktop and then you used your iPad pencil on the iPad, different things like squares would come out of your pencil instead of just a line. Yeah, yeah, exactly. So the basic idea was the desktop environment had a visual programming environment in it
Starting point is 01:01:37 that gets compared both to Scratch a lot, and it probably owes more to Scratch than anything else, but also gets compared a lot to node-based data flow programming like Max, MSP, or Grasshopper. And I would say it's actually what you're creating are state diagrams. So it's a state and constraints model, which... State charts? Yeah, yeah. We looked a lot at Steve Oney's Interstate as a way of informing the design of this tool. So it's technically not the same thing as the way the Dataflow languages in Grasshopper work, but it looks similar in that transitions are event-driven. It's similar in some ways to Macs in that regard. Then the iPad environment looks kind of like a really simple version of a tool like Procreate or some other tablet-based bitmap drawing environment.
Starting point is 01:02:27 But instead of having like a tool palette that is defined up front, the tool palette is whatever programs that you create in the programming environment, and they're linked. And you said bitmap. We're not doing vector graphics anymore. No, and the reason we did bitmap was really because we started earlier with dynamic brushes, testing it with artists. And we were working with people who had worked primarily in a digital painting scenario and they were used to bitmap tools. So they wanted opacity. They wanted
Starting point is 01:02:56 the ability to feather the edges. You can do that with vector graphics. And there's a version of dynamic brushes that is completely vector, but we haven't returned to that but it was more just the audience that we were looking at there's no reason it couldn't be a vector tool necessarily but we chose to align it with bitmap drawing because that corresponded with the types of people that started using the system early on cool so yeah i think i interrupted you you were going to talk about some of the programming model stuff, the interesting bits. Yeah, yeah.
Starting point is 01:03:29 It's so funny. I usually give this with some images behind me. So it's a useful practice to describe it without images. So the kind of core, I find I, I guess I often design my languages around three primary programming concepts or components. And so in Para, it was constraints, lists, and declarative duplication. And in Dynamics... Why three?
Starting point is 01:03:52 Ah, it's like the magic number. I don't know. I guess, arguably, in Dynamic Brushes, you could say it's four. But it's primarily data bindings, so just declarative mappings between some input data, most commonly the stylus, but then you also have the synthesized data, like function generators, like a sine wave. And then you also can import in external data. We also added the ability to use mic input, the accelerometer of the tablet. can be mapped to a brush property. And the properties are obvious things like position and color and opacity, but then also less obvious, but very powerful things like geometric transformation.
Starting point is 01:04:32 So rotation, scaling, and then you describe data bindings inside of states. So that's the second thing. And then this is basically whenever a state is active, the data bindings within it are active. And you can only have one state active at a time, just, you know, traditional notion of state
Starting point is 01:04:52 machines. And transitions between states are driven by these event-driven transitions that we predefined in the first version. So you have a limited set of events, kind of similar to Scratch, like stylus events, when the stylus is down, when the stylus is up, or when a certain interval of time has elapsed, or when a certain interval of distance has been traversed. And then we're working on a new version, which has a more expressive event definition structure, so you can describe more arbitrary events or evaluate arbitrary conditions. Yeah, so states, bindings, transitions, and I guess the fourth thing would be then there is also a small set of actions, which are just very simple methods that are called once, and they can be called once on any transition that occurs. For example, you can initialize a new stroke when a transition occurs, or you can start a timer, which is really useful for doing time-driven transitions. Or you can also call this action called spawn,
Starting point is 01:05:53 which is the most powerful, but maybe most complex aspect of the system, which basically lets one brush generate another brush or multiple instances of another brush and so the reason we added spawning to the system was because of what i was talking about several minutes ago where we wanted the ability for the artist to create brushes where multiple things were happening simultaneously you had one input and multiple outputs or you were drawing on the canvas but had some automated things that would be drawn in response to what you were drawing. And so spawning basically lets one brush generate copies of itself or brushes generate automatically that draw on a timer interval as opposed to being driven by changes to the stylus input. The example we were really trying to build up to was this drip brush, where as you draw a line, drips automatically come off of the line
Starting point is 01:06:45 and fall down across the canvas. But yeah, there's been other work we've done in that space. Like you can generate really complex radial forms where you're just doing one input and then you automatically have multiple radial branching forms coming out from that one input. It's a lot easier to show with videos.
Starting point is 01:07:02 In a prototype version, we added the ability to actually do recursion where a brush could spawn show with videos. In a prototype version, we added the ability to actually do recursion where a brush could spawn instances of itself. And then we had a termination condition based on the number of brushes that have been spawned. We didn't actually use that in the final version of the software because recursion can generate some tricky debugging scenarios. And so we're still working on refining that part of the system but that's the overall idea yeah i i don't know why it just occurred to me that i came up with a way to extend the level of abstraction here like in it's probably already occurred to you you just take this but bring it into um like a vr 3d world where instead of making
Starting point is 01:07:42 brushes you're making like clay basically it's like a clay making kit and so you can like make clay over here like you quote mix the clay programmatically with blocks and then you like walk over or maybe one person's mixing clay the other person's like shaping the clay so it's like it's like the same thing but for sculpting yeah that's interesting i've talked with people about doing stuff in vr yeah more in a 3d application we chose 2d originally because i love 2d art and also it's a lot of work to to build these systems and any any corners you can cut in terms of simplifying the things you can build is is a good starting point um but uh, we've talked about doing a 3D version of
Starting point is 01:08:27 it. I'm actually really interested in applying it, and we have done a little bit of experimenting in this space already, to forms of... So one way of thinking about this is, I kind of framed it in the lens of tool development. You're building your own tools. But another way of thinking about it is reframing drawing within the space of interaction design. So instead of actually making the artifact, what you're doing is defining the interactions you want to have with the system that's going to then generate an artifact. That idea is something I was thinking about or kind of came to me from these conversations I've been having with the artist and engineer Madeline Gannon, who is probably best known for the gestural robotics
Starting point is 01:09:14 control she's done with these industrial scale robot arms. But she's actually doing all this work from a standpoint of what are different ways in which we can think about digital fabrication interaction and control and what are the ways in which we might think about CAD computer design not as being this process of sitting in front of a computer and creating a design and then sending the machine to be fabricated but more of this process that looks more like what happens when we make an object in a more traditional method like she's really interested in on-body fabrication and clothing manufacturing, where you actually might design an object while you're working on a mannequin
Starting point is 01:09:50 or actually working with the person themselves. And so she has this really powerful idea of, can we think about interacting with some autonomous agent, not as upfront we describe what it's going to do with the artifact, and then that's translated to tool paths, and then the system executes it. But instead of saying, like,
Starting point is 01:10:09 what are the class of interactions we might want to have, or what are the ways we want this autonomous agent to respond to us as we work, and let's put the designer in the position of describing those, and then being in the context of actually working with this machine or autonomous system or agent to collaboratively, or I don't know if collaboration is the right word. Metaphors are important here.
Starting point is 01:10:32 I'm not sure I'm choosing the right ones, but to finish the artifact. So I've been thinking more about dynamic brushes from the standpoint of what would it mean if we're describing not how a digital brush is performing, but how the head of a CNC machine or a 3D printer is moving in response to decisions we make in the moment of digital fabrication as opposed to ahead of time.
Starting point is 01:10:55 It maybe seems like a big jump, but we've already done a version of dynamic brushes where we've connected it to a CNC machine and used it to create brushes that actually draw in large format on the CNC machine and used it to create brushes that actually draw in large format on the CNC machine. The transcript for this episode is brought to you by Replit. Normally, when I do a Replit sponsorship, I spend some time reflecting on the features
Starting point is 01:11:22 of Replit that I am most personally excited by and try to write up something cohesive and organized so that I don't spend a bunch of time kind of bumbling and making an ass of myself. But today I'm actually going to do that because I wanted to relate an anecdote from earlier this decade, and it felt better to do that just off the cuff rather than writing something out. In 2016, my wife and I, having just recently been married, decided to tour around Europe, living out of our backpacks, as is sort of the typical thing that people from North America do when they get married. I spent a considerable amount of the trip trying to learn Cloj learn closure because that was a language that I'd recently become quite infatuated with and hadn't yet cracked my knuckles and done
Starting point is 01:12:12 a big project with it. So this felt like the right opportunity to do that since I'd be spending a lot of time in hotels or riding on trains without much else to occupy me. The experience of setting up Clojure turned what would have been a very exciting point in my life, where I get to dive into a new language, try a bunch of projects, and make a bunch of cool things into several weeks of protracted misery. There were multiple different build tools, and as somebody coming from outside the Java ecosystem, I had no idea what palm meant or what maven was. I had to spend a lot of time learning all of this terminology, installing all of these different tools,
Starting point is 01:12:55 and making configuration file after configuration file, trying to get it to work. And after weeks of just trying to get Clojure set up, I finally had something that let me actually build the project I was excited about, which, for the real heads in the audience, was the first version of HEST. That early experience with Clojure colored my impression of the language and the ecosystem in an irreparably harmed way. And I never ended up really embracing Clojure the way that I originally thought that I was going to, and I've since basically stopped using it entirely. So where does Replit come into this? On Replit, you can be up and writing Clojure in a matter of
Starting point is 01:13:41 seconds. There's nothing to install, There's nothing to configure. You just tell Replit that you want to write Clojure, and then you write Clojure. If I had had that in 2016, my relationship with Clojure may not have been adversarial right from the first time, and I might still be writing Clojure, because it's a beautiful language. It is a wonderful work of engineering, but for whatever reason, all of the design that has gone into it in terms of the language primitives and the way that the technical decisions are grounded in solid conceptual design work never manifested in the beginner experience. Replit did the work that the Clojure core team should have done. So this is my way of sincerely thanking Replit
Starting point is 01:14:29 for creating something that solves a real problem for newcomers to programming. And that's why I mean it when I say thanks to Replit for bringing us the future of coding. I was thinking about how you said that you have like electronic circuits but now they're also they like have a dual purpose and i was thinking about um the like part of what's great about gestures is the embodied knowledge that we have but also just the joy of moving our hands in ways that aren't just twitching our fingertips. It's just more joyful to move our whole arm.
Starting point is 01:15:08 But that's just our arm. So I was imagining one day it's going to be like you're just going to dance. And the way you move your whole body is going to be somehow mapped to some other creative form of expression that might be a physical object. Yeah. If you want it. You could design it so that the more you bounce more you uh bounce like the i don't know the shakier you know you could i like the idea of allowing people to design their own interactions based on how they have embodied the thing they want to express yeah i i i'm really excited by that too i think this also gets to
Starting point is 01:15:44 the heart of like the the the challenges I'm most interested in addressing right now and so one of the powerful things about programming is you can create these arbitrary mappings between any input to any output you know within within reason but um and that was actually one of the really nice things that came out of doing these studies with artists with the dynamic brushes system is actually had this moment of observing this painter have that realization where he was working with the system early on. He was saying, I don't understand why you, you know, every drawing tool I've worked with you, I'm paraphrasing here, the stylus position dictates the brush position. Like, why would you ever not want to map stylus position to brush position? And then at one point he said, oh, oh, okay. Actually, like I can transform the stylus position in some ways where I can use it to dynamically control the speed of the brush in addition to where it's actually located.
Starting point is 01:16:42 Or I can have it so that the position changes the color of the brush that's interesting oh it's totally arbitrary how I map these things oh every tool that I've used those relationships are arbitrary in in the space of software and oh I can be a wizard I don't have to like take pigment and spread it around with matter. I'm a wizard. I can have it do what I want. So that was really cool to see because, yeah, I think that's one thing I'm super excited about in helping people to create their own software and tools with programming is you can think differently about these relationships and these mappings. But it's also, the hard thing is then there are all these wonderful things about when you're getting to the space of mapping data from interactions that are tacit or embodied or gestural, there are all these understandings that people have about
Starting point is 01:17:38 that, the way they move through space, the way they draw, that to do really powerful or specific or precise things with them in a symbolic context, it can be very useful to understand them from a different perspective, from a more discrete perspective, from a more formal perspective. So artists think about, for example, the way they draw as a time-based act, many artists I've talked with, but that doesn't mean that they think about it in the way that dynamic brushes represents time with regards to drawing. That doesn't mean that they think about, you know, it takes me approximately 10 seconds to move this distance and I want to have this, you know, behavior in my drawing execute on a certain time interval. There's a non-trivial process that goes between this
Starting point is 01:18:26 embodied or tacit understanding of drawing as a time-based process to how can I symbolically manipulate that to do the thing that I want same goes for physics and I think I'm always sensitive to the fact that on one hand there's a tension there and there's a chance for trade-offs or there's a you know i i don't think you can preserve everything you get from one way of knowing like a tacit way of knowing when you move into a symbolic or formal representation but i do think there's power and utility there. So I, I, I, sorry, I just want to, this is something that I was gonna bring up my myself in a few minutes. I'm glad we got here. Now. What don't you think is possible preserving your? Yeah, which which one can't you go into the
Starting point is 01:19:21 other? Yeah, sorry, I just missed it. I think it's both ways, right? You can't transmit from one to the other without losing something in the middle. They're like... Yeah, and this is maybe getting too abstract, but any translation is an exercise in losing something, right? There's value in it, but... I guess, no, technically, you could translate things losslessly.
Starting point is 01:19:51 I would say like from the perspective of, I've been trying to write about this lately and it's been very challenging. So I want to choose my words carefully here. One way of thinking about it is, so, so earlier I talked about how drawing is this process of being a often speed and, and not thinking is an important element of drawing where you basically artists train themselves to build up this physical intuition so that they're not explicitly analyzing everything that they're doing and instead reacting. Like people who play sports.
Starting point is 01:20:32 And maybe, you know, I've also observed programmers and people have talked about this where they've reached such a familiarity and level of skill with programming that what they do is almost reactionary or intuitive in some senses when they're, when they're developing some things. But I would say my experience with programming is in general, it's this process of analysis, of taking an idea and breaking it into smaller achievable problems.
Starting point is 01:21:03 And that is very different from this notion of reacting in the moment. Almost system one, system two, the Kahneman's distinction. Yeah, maybe. I want to be careful. There's been a lot of work in the psychology of psychology of programming and also psychology and and psychological work and theoretical work around tacit and embodied forms of working and I I would say my work could benefit by drawing more heavily on that space although I am really inspired by things like the textility of making or the reflective practitioner. Those works really drive how I think a lot about some forms of expression that I'm interested in supporting.
Starting point is 01:21:58 I recognize the tension in my own work between trying to combine something that is often a fundamentally tacit and embodied form of creation with something that is often most powerful as an analytical form of creation, which I think programming is in many respects. And that there are different pathways
Starting point is 01:22:22 you can take in that process. The pathway I've taken is to try and select programming representations that preserve some really important expressive aspects of the passive form of creation like drawing. But I also recognize that they're going to shift some of the things that people are doing in that space. So dynamic brushes, I never have claimed that it's something where people can just automatically learn the programming language and start creating tools and drawing like they were before. No, like they spend time
Starting point is 01:22:52 learning the language. They spend a lot of time authoring tools, which they wouldn't be doing necessarily with other systems. And there's more labor placed in that space of learning symbolic programming and writing and debugging the tools they're building and also understanding what are the trade-offs in that space. Another approach might be to build a really expressive set of procedural drawing tools, give people the ability to tune some parameters of those tools, but don't have them invest in the actual process of writing the code itself. And there would be a lot of benefit to that approach. And there have been things that people have done in that space.
Starting point is 01:23:29 But I think now I'm much more focused on this idea of, okay, recognizing these trade-offs, recognizing that you have to think about drawing numerically. You have to think about it analytically. You have to think about it systematically. What's the way in which we can help people make that transition more easily? Most of the work we're doing with dynamic brushes right now is really around this idea of how can we show people representations of the drawing input they're using in ways that help them to understand why a tool is functioning the way that it is, why a state change occurs when it does, and help them make
Starting point is 01:24:05 informed decisions about the process of programming. Yeah, it's a trade-off. Yeah, I definitely see the trade-off of, I do a lot of thinking in this area too, but I'm going to be a little bit less careful with my words. I'm just going to like kind of talk around it to evoke what I'm getting at i guess i see on the one hand there's things that humans already know like we have a very embodied sense of gravity we're not born with it but i think over the first five years like a pig kind of way like we get it you know like oh okay like i drop things i expect in the fall it's like you feel in your body and you feel it and you expect it in objects you manipulate but then it takes a long time a lot of formal training to get someone to transfer that
Starting point is 01:24:50 to a knowledge of physics like an abstract knowledge of physics there's like it's it's quite a quite a lot of education that has to go to teach that formally and i guess maybe a better example one that's kind of close to my heart, is I was a logo kid. I was a math phobe who was exposed to logo. And I learned how to draw a circle. I kind of eventually intuited on my own, oh, wait, if I just walk up a little bit and turn a little bit and do that 360 times, I got a circle. Then when I eventually went to calculus, it was just so easy to understand what a derivative was because I've walked along curves before. Anyways, what I'm getting at is, one way to put it, is Alan Kay has this critique of Scratch that you may have heard of.
Starting point is 01:25:34 Yeah, yeah. I guess you haven't worked on Scratch, but I feel like you're maybe one of the better people I've had on the podcast who could defend it from Alan's criticism. Basically, there are certain human universals or things that humans are good at like drawing and singing and dancing and playing but then there are certain non-human universals that are not more enlightened but just they're also very desirable
Starting point is 01:25:56 but they don't come naturally they require a certain kind of investment but then there are these interesting ways to get humans from the universals, the things they already know and love and like transfer leverage that love and knowledge into a new, more powerful, but also harder domain. So anyways, I'll let you take it from there. Yeah, I mean, that makes me think of, and maybe this is what you're drawing from, like Alan Kay's essay on, I think it's a user interface, a personal view. I'm not sure if I'm getting that right, where he talks about this notion of
Starting point is 01:26:28 how you can move between more concrete forms of understanding to more symbolic forms of understanding and symbolic knowledge versus visual knowledge. This is coming, I think, from Jerome Bruner's theories about different forms of knowledge, and I'm not representing it in its entirety but this idea that you might be able to yes that like symbolic knowledge being this really powerful form of representation but maybe less natural for people to understand but visual and kinesthetic knowledge being more familiar and more natural but that by working in the visual and kinesthetic domain you can help scaffold the process of understanding and manipulating symbolic knowledge i'm really excited by that idea and i think it's something that i would say i need to do a better
Starting point is 01:27:14 job with some of the systems that i do in thinking about so much focus initially has been on how do we select an appropriate representation for a given domain with still the knowledge of there's there's going to be learning challenges or conceptual challenges here and what are the ways in which we support people in making connections or or moving to really understanding the symbolic domain when they're starting with a visual or kinesthetic form of manipulation. And I think there's a lot that I can do, not hiding information, showing information visually. There's also a lot of huge challenges. So one of the very concrete challenges in dynamic brushes
Starting point is 01:27:58 is we have people working in a geometric domain, 2D drawing. And then we have this type of data that is geometric. It has natural geometric properties, the stylus input in many ways. And we have these other data types that you can map them to geometric properties, and it's very powerful, but they're more abstract. They don't inherently have a geometric quality that aligns with the geometric properties of the 2D canvas. So like a sine waveform. I can draw a sine waveform on the canvas, but that's also misleading in some respects because the dimensions of it are going to be arbitrary
Starting point is 01:28:32 relative to my drawing. And really that data is, it's also very powerful to understand that data more abstractly or numerically. And so there's this idea of, okay, yes, I want to help people draw those connections. I also don't want to mislead them. I don't want them to have an incorrect intuition about how to think about this type of input and this data. It would be a problem if the painter got the idea that,
Starting point is 01:28:57 oh, there are certain types of relationships I'm supposed to create with data in dynamic brushes and other types that I'm not supposed to create. So I don't think I have answers to that, but it's a topic I'm really interested in. One of the big challenges I see in that is the more programmatic of an interface you expose, the less constancy or repetitiveness they can expect. So the harder it is for them to build up those somatic intuitions that are so important. Part of what makes, allows artists to get really good with a paintbrush
Starting point is 01:29:28 is that paintbrushes are pretty similar, but you're allowing them to create all these new paintbrushes. So I would expect, or one solution to that problem would be they spend a lot of time iterating on what paint brushes or what uh you know uh dynamic brushes they want to make and then they kind of maybe will leave it kind of like it is and then make a bunch of art and then the next series they'd like modify the brushes again because if you modify the brushes too much it's i could imagine that it's hard to build up a certain intuition of like what different gestures actually accomplish yeah and that's the behavior that we've observed so far with people using the system is
Starting point is 01:30:10 there we'll spend a lot of time in the programming environment kind of experiment drawing and then a lot of time in the drawing environment and really only making minor tweaks if anything to the brushes but i'm not saying that's wrong I also think that we're working hard to find ways to enable people to move more easily back and forth between the two spaces because there are ideas that emerge during the process of drawing that you want to be able to implement in in programming and to be able to translate to the programming environment it's not always easy so we've done really simple, obvious things like you can record some drawing input and loop it back through the system while you modify the code.
Starting point is 01:30:50 So you don't have to like experiment every time you can immediately see how your changes update the drawing. But there are, there are bigger questions that I know have been explored by other people in other domains, but this idea of, Oh, do I ever want to be able to, to do something in the drawing environment that actually describes or manipulates a relationship in the program? And Brett has done some really cool stuff in this space. of how do I distinguish between what I could do that's possible versus what makes sense and what has like a meaningful semantics. You know, like I love the idea of having people be able to draw a waveform
Starting point is 01:31:35 in the drawing environment. And then, you know, that's a great idea. You know, less clear, like I've talked about this a lot with people on the the scratch team around like how would you geometrically describe a conditional or not you know you could come up with an idea and there might be some cases where it makes sense but there might be a lot of other cases where it might not make sense so yeah a lot of challenges in this space one of the things that makes this very concrete for me this idea of like a standard interface versus a programmable interface
Starting point is 01:32:13 is um 10 years ago my dad got this like whole new tv system and it came with this touchscreen interface and he could never turn on let alone use this tv and the newer interface like the state of the art of today is made by the same company but it's a um it's not a touchscreen anymore it's like back to an old-fashioned interface where you actually specify to the company what you want printed on the buttons and they print it onto the buttons uh so it's like the same amount of programmatic programmability but um but once you've set it up they print it onto the buttons. So it's like the same amount of programmability, but once you've set it up, they print it in a way that makes it much more like my dad can not only can turn on his TV now, but he doesn't have to look at the remote to know where the
Starting point is 01:32:56 buttons are. That's interesting. Yeah. Yeah. It's, I think there's need for more work in the domain of creativity support and systems research. I think looking at some more longitudinal impacts of use patterns and learnability for some of the tools we build over time. Like, I think there's, in general, this idea that people should learn a system really quickly or use it really quickly, and if they can, then it's a good system, and if they can't, then it's a not effective system. But that's not necessarily the right metric for how we think about how we might support people, you know people using software or using a programming language or a creative tool. And some of my favorite things come out of looking at
Starting point is 01:33:51 what artists who have been using Photoshop and Illustrator for a really long time do with it, where they use the filters in ways that I had no idea that you could use them in those ways. And they'll call them hacks. They'll be like, oh, I use this hack. It's not really how it's supposed to be used, but I get this effect by combining the blur filter with the, I don't know, I'm not going to remember the filters. I never
Starting point is 01:34:14 use them. And it's interesting because it's like, oh, you could see that as an unintended use, or you could see that as this is what happens when people use software of an extended period of time, they adapt it to their own applications. And that's a really powerful and valuable thing, but it's really reflected in how we present and evaluate systems from a research perspective. And that it's a system we build over a year to six month period, and we test it for hopefully a while. And then we kind of understand what people can do with it in that timeframe. But not what people might do with it using it over a year or two years. There's only exceptions to this, but.
Starting point is 01:34:53 I have a very funny, I guess, illustration of this kind of dialectic. I've been having this silly conversation with a stranger on Twitter. His position is that programming should be as intuitive as possible. And my position is that it's not necessarily a bad thing if it requires certain investment,
Starting point is 01:35:09 but that investment pays off in power down the road. Yeah. And so he sent me yesterday a video of a monkey, like an actual ape, using a touchscreen. And he's like, see, programming should be this intuitive that even a monkey can use it. And so I sent back a video, like a timeless video video of someone who didn't know how to play piano. At the end of the year, he could play really intense piano.
Starting point is 01:35:31 It's like, no, no, this is what programming should be like. And I think it's a funny dichotomy of the two different views of immediately usable versus rewards investment. Yeah, I guess if you think of programming from the interface or environment perspective, yes, we should make things easy with an important asterisk there, which is like easy without restricting the important forms of power expression that we want to deliver. But if you think of programming as a way of thinking, it is almost like not relevant to think about like, how do we make this way of thinking easy? Because it's more about, you know, we want to provide pathways
Starting point is 01:36:10 into engaging this way of thinking, but you can't trivialize the difficulty of learning abstraction. You know, I remember when I first really understood the concept of object-oriented programming, it's a powerful idea. And I don't think you want to say, well, there are things that are hard about object-oriented programming. Let's choose a simpler idea that's more intuitive, more obvious to people. And you're diminishing the power of that idea, you know, and arguably like object-oriented programming is more aligned with how people might think about things in other programming models, but there's value in understanding and learning different ways of seeing the world,
Starting point is 01:36:52 and programming offers that. I don't want to diminish that. And also, the idea of what is intuitive, I use it a lot, so guilty of this, but we don't really have a good grasp of, I think, in many ways, things that are intuitive, Because what's intuitive for one person could be totally different from what's intuitive to another person. And the notion of dynamic brushes, we chose this representation for drawing not because we thought, oh, this will be the most intuitive representation for drawing. Because this is how people think about drawing. Because every artist I talked to thought about drawing in a different way. There were some similarities that we tried to recognize, but overall the idea was,
Starting point is 01:37:29 no, if I build a programming representation for how I think about drawing, it's not going to correspond with how this other person thinks about it. It's an impossible design challenge. So I think there are other principles we can involve in terms of what does this let people express, what does it limit them from expressing, and what are the reasonable scaffolds we can provide to helping them learn it. We want to think about things in terms of correspondence and how close the mental model of a given audience is to a given representation, but that's not always feasible. And I would argue in programming,
Starting point is 01:38:08 it's often not feasible. Oh yeah. I'm glad. I'm glad we, I don't know. I like, I like that answer a lot. Um, uh, and that, I didn't know if that's what you were going to say. Um, I, I thought maybe you'd come out on the other side of like, no, no, no, it has to be as easy and natural as possible. Yeah. Easy as possible, but no easier. And I'm ripping off multiple people with that statement.
Starting point is 01:38:33 So back to some of your work, I liked your discussion of the evaluation of it. You talked about how tricky it was. And I think it's kind of, it's similar to whenever you make a tool, it's hard to evaluate a tool because you have to evaluate how people use a tool and what artifacts they produce. So it's like, we are one step removed in a way. But I think for a creative tool, it's even harder. Like, so I'm someone who designs tools for people to build user interfaces. And so I, there's this project called seven GUIs where there are seven user interfaces and I just can watch people build those seven interfaces in my tool and call it and call it a day. But, um, when you're building a tool for artistic, uh, creative expression, you're, you're building a tool for people to like purposefully break
Starting point is 01:39:18 boundaries and do things that have never been done before. So's like so it seems really hard to evaluate that sort of a tool yeah it is hard uh so one of the the best metrics that i think mitch and leah both held up and i've similarly tried to pursue this metric in my work is like you know a system is successful when you see multiple people making things that you didn't know were possible in that system it's great if if if people are able to make complex things or able to easily do the things that you thought the system would be good for but if someone uses it to then do something you had no idea was possible then you've succeeded in some way from a creative standpoint. How you design for that is not obvious. I think you just designed with a blindfold on. I couldn't have imagined they did that.
Starting point is 01:40:16 I wasn't looking. Yeah. Jay Silver has this notion of developing a tool around a project space. So if you develop a tool that's able to create one type of project, then you've got a point. If you know that the tool can create these two types of projects, then you've got a line and you've got a spectrum of projects across that line. If you know a tool is able to create these three types of projects, then you've got a space and you've got a larger area of possible projects that might emerge within these outer parameters. And he describes this in his dissertation. And I think that's sort of one way of approaching both the design and then
Starting point is 01:40:56 evaluation, where you set up, and a lot of the work that I've done is sort of gravitated toward this, you try to choose these targets of the types of expression you think your tool should be good for, but pick them in a way that they are different enough that you're hopeful that there will be some interesting variation in between them. You design for those, you inform that through different processes, looking at what people do while you're building it,
Starting point is 01:41:20 and so on. And then when you evaluate it, you also think about, who am I selecting that corresponds with or diverges from these forms of expression that I've tried to design for and then looking at what they do and how it aligns with these points
Starting point is 01:41:36 you've kind of created versus unknown areas or points between them and really trying to look at both what you've set up as sort of the types of things you think the system enables making and also what people make with it and how close or how far it is from those things. So that's one reason that I've also tried to have people use the tools that I've built over a longer period of time is because you get more interesting artifacts out of that and you have more data with regards to the forms of expression that are
Starting point is 01:42:05 possible by analyzing those artifacts. It's definitely a qualitative method of evaluation, and there's a lot of noise in the data, so there are limitations to it. I recognize that. One thing that I'm trying to move towards is actually scaling up some of these studies, and so we're working right now on releasing dynamic brushes as a stable app for people to use, although that's a long process so that we can look at more generally, what do people do with the system independent of a formal study or in the wild, quote unquote. And that's really inspired by some work that Leah and Benjamin Mako Hill, who's professor at UW, did with the LilyPad,
Starting point is 01:42:47 where they looked at what people were doing with it over the long term in actual practice by looking at sales data and work posted to different online communities and doing more of a longitudinal and data-driven approach to analyzing what are the creative affordances of this specific platform. But I'm always interested in other forms of evaluation,
Starting point is 01:43:09 but I think that fundamentally, if you're designing for creative expression, you need to accept that there are trade-offs in that traditional metrics for evaluation, efficiency, speed. Those are great, but they won't work. They won't tell you the whole story here. Yeah, and that
Starting point is 01:43:25 was one of my questions can can people use these tools exciting to hear that they'll be coming soon yeah so para might be coming in a different version than than our original prototype dynamic brushes we've been building it we're working on another version that has these debugging support features in it my goal is to get in the app store with the the release of that work um it's another challenge of of academic research is it's often not incentivized to release your software as important as that is so i'm trying to find my own strategies for for doing doing that. Um, often the paper is the artifact. So. Yeah. Yeah. Well, I guess this is kind of, uh, dovetails with another question I have the, the, the, um, uh, interplayedness, interplayed, interdisciplinary, interdisciplinary.
Starting point is 01:44:16 That's the one. Nope. I'm just going to skip the words. The systems engineering method that you did where you do this need finding research and then you build a tool and you watch people use it you evaluate it then you kind of repeat and build a new tool it feels a little bit rare to me like i didn't think that academia would incentivize this kind of work it seems like academia it doesn't seem to support need finding as much i'm kind of I was excited to see that that was like a part of your process. Yeah, it does and it doesn't. I mean, I've been fortunate to work primarily in the field of human computer interaction where it's a big enough community
Starting point is 01:44:58 with a diversity of thought. So there are definitely people who are more interested in really quantitative evaluation, forms of measurement that are less noisy. But there are also huge groups of people who really value tackling messy problems and also approaching it from a human-centered design perspective, which need finding is a big component. These things take time. Building software takes time. Testing it with people over a longer period of time takes time and resources. Yes, that's not always supported in an academic context, but I'm also encouraged by the fact that there are increasingly in computer science research emphasis on things like open source or verified artifacts or functioning tools as a contribution and as something that we officially recognize.
Starting point is 01:45:55 And so my hope is that there will be more official incentives for doing this work in addition to the knowledge gain that we can get from it. And yeah, I'm also excited to see as I continue in my academic career running a lab as opposed to working as a student, what opportunities I'll have for more collaborative forms of production and trying to scale up projects over a longer period of time and with more people involved in them. What kind of like Scratch, I guess, is like the big example? Scratch, I think, could only happen at the Media Lab. Scratch is a very unique example in that it's basically a small company being run inside a research group. And I think it's unlikely that I would...
Starting point is 01:46:41 Could pull that off, you know. Yeah, like Mitch has made really good choices in that respect but he has you know made a decision to um like he wants to get programming to as many children as possible which I think is a really laudable goal and and scratch is a an approach to doing that I think I'm maybe more leaning towards one of the benefits of academia in that you can explore many different directions. And so I want to release software
Starting point is 01:47:13 and I want to think about building systems that have a lifespan beyond the paper. But there are many ways to do that. For example, I've had great partnerships with Adobe and industry, and I'm really interested in pursuing those going forward. There's trade-offs due to that, but so far that's been a good option for me. And there's also opportunities to contribute to other creative coding communities that are more focused on specifically putting
Starting point is 01:47:41 systems out in the world. And so the work that the P5.js and the processing community and also the open frameworks community does is incredibly inspiring and valuable. And I also think about ways in which in academia, I might be able to, in computer science academia, make more connections between the work that's being done in the creative coding community and the work that's being done at conferences like CHI and human computer interaction research and programming languages research and how there can be an exchange of ideas but also an exchange of resources in some respects. Cool. Well, I think we have taken enough of your time. No, this is great. I really enjoy hashing through these ideas.
Starting point is 01:48:31 Clearly, I have a lot to continue to hash through, but it's been really fun to talk with you about this. Yeah, yeah, it's been great having you. Yeah, definitely. Love being connected with new people, thinking about similar ideas. And it's always encouraging to talk with people who are interested and value the notion that programming can be
Starting point is 01:48:48 better by building different programming languages or different types of programming. We can open it up to new groups and new people. That's not always an idea that everyone believes in. It's good to be reminded of it.
Starting point is 01:49:03 Before we sign off, I wanted to give you an opportunity to just list some of those places on the Internet where you could be reached in different ways that you wanted to be reached if you were looking for collaborators or anything like that. Yeah. So I guess very concretely, if any of these ideas are exciting to people, I'm starting up a research group at the University of California, Santa Barbara in July as an assistant professor. And so we are actively looking for people who are interested in potentially pursuing these ideas under a PhD or a master's degree, but also visiting researchers, visiting artists. So if you're excited about this and you're interested in talking more, definitely reach out to me over email, jmjacobs at ucsb.edu. But you can also find me on my website, which is just jenniferjacobs.co. Yeah, I think that's probably the best way to talk with me, but I'm also on Twitter at JSquare. That's it for our show today. to steve for having the foresight to interview jennifer jacobs i was actually gearing up to reach out to her and ask her to come on the show myself when i learned
Starting point is 01:50:17 that you had already done this so that was very fortuitous and i um i was delighted to hear this interview there's a not insignificant chance that I will probably at some point ask Jennifer to come back on the show again just so that I can ask her a bunch of questions that occurred to me while listening to this show. That's the real benefit of having a podcast like this is that I get to reach out
Starting point is 01:50:41 to all of these interesting thinkers in our field and ask them whatever I want to ask them. And so on that note, if there are people out there that you would like to hear me interview, I have a fairly long list of guests that I've selected already. And so not all of your recommendations will be people that I can eventually bring on, but there are many people who would be a perfect fit for the show that I've never even heard of. And so if there's anybody out there that you know of,
Starting point is 01:51:14 and I'm just going to explicitly say extra, extra points if they are from an underrepresented group, since I am very committed to having this show reflect those principles of inclusion and representation that I spoke to earlier, please tell me who they are so that I can bring them on and share with you all of their wonderful thoughts and visions for the future and wild ideas. The next episode is going to be an interview with Ravi Chung,
Starting point is 01:51:43 one of the key figures in the Sketch and Sketch project, which Jennifer mentioned in this interview and which has made waves through our community in the past. Ravi and I had a very long interview. It was something like three hours and 45 minutes. So I'm either going to cut it down significantly or split it into a two-part. So this one might take me a little while yet to finish editing and get out. So if there's a bit of a delay between what you're hearing now and when the next episode is released, that's why. You can find the show notes at futureofcoding.org slash episodes slash 48. You can join our Slack community at futureofcoding.org slash community. I'm Ivan Rees, and I will see you in the future. Продолжение следует...

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.