Future of Coding - Jennifer Jacobs: Para & Dynamic Brushes
Episode Date: June 14, 2020"Metaphors are important here." There's a small handful of people that I've been requested again and again to interview on the Future of Coding podcast. Jennifer Jacobs is one of those people. Her wor...k on Dynamic Brushes in particular, and parametric drawing in general, occupies a major intersection between disciplines and provides insights that we can all apply to our own work. This interview touches on childhood education, programming tools for both non-programmers and expert programmers, tangible interfaces, wearable and embodied computation, aesthetics, the relationship between academia and industry, means of evaluating the efficacy of projects, geometric encodings of first-order logic, symbolic representations, whether Scratch could exist outside MIT, and more. Jennifer does a wonderful job articulating the nature her own work, but also the works of her collaborators, peers, and influences, so that we come away with a great understanding for the broader spaces in which her research fits. Jennifer is already am important figure in our Future of Coding field, and I am very excited to follow her career and see all the places the impacts of her work will be felt. You'll notice right away that Steve is sitting in the interviewer chair this time. This is the first of a handful of episodes that Steve recorded in 2019 but didn't release. I'm planning to edit and release them throughout 2020, so you'll hear a bit more of Steve yet. The transcript for this episode was sponsored by Repl.it. Show notes and the full transcript are available here: https://futureofcoding.org/episodes/48Support us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.
Transcript
Discussion (0)
Just writing down what I think might be a good title for the podcast,
designing tools for creative expression.
Yeah, that's a reasonable title, yeah.
Welcome to the Future of Coding.
I'm Ivan Rees.
There's a small handful of people that I've been requested again and again to interview on Future of Coding podcast.
Jennifer Jacobs is one of those people.
Her work on dynamic brushes in particular and parametric drawing in general occupies a major intersection between disciplines and provides insights that we can all apply to our own work. This interview touches on childhood education,
programming tools for both non-programmers and expert programmers, tangible interfaces,
wearable and embodied computation, aesthetics, the relationship between academia and industry,
means of evaluating the efficacy of projects, geometric encodings of first-order logic,
symbolic representations, whether Scratch
could exist outside MIT, and more. Jennifer does a wonderful job articulating the nature of her own
work, but also the works of her collaborators, peers, and influences, so that we come away with
a great understanding for the broader spaces in which her research fits. Jennifer is already an important
figure in our future of coding field, and I am very excited to follow her career and see all
the places the impacts of her work will be felt. You'll notice right away that Steve is sitting in
the interviewer chair this time. This is the first of a handful of episodes that he recorded in 2019
but didn't release.
I'm planning to edit and release them throughout 2020, so you'll hear a bit more of Steve yet.
The show notes for this episode are full of links to all the various projects and collaborators and inspirations that Jennifer references in the interview,
along with a video of her demoing Para at the Adobe Max 2014
conference. You can find those show notes at futureofcoding.org slash episodes slash 48.
There's also a full transcript if you'd prefer to read this interview rather than listen to it.
One final note before the interview starts. As the host of this podcast and the steward of
our community, I want to say something in my official capacity. Black lives matter. The
consequences of racism in the U.S., here where I am in Canada, and around the world permeate every
facet of life and society. It will take a constant effort over many lifetimes to achieve justice, and all
of us can play a role in making that happen. There are many ways we technologists can be deliberate
in our own work to be sure we are empowering the people who need it most. Right now, I'm just going
to look at one small piece, our community. Over the past six months, we've had a number of deep
probing discussions about the need for greater diversity in our membership, in the guests that appear on this podcast, and in computing broadly.
One outcome of those discussions has been our code of conduct, which I'd like to read a few sentences from.
One of many principles held by the community is that computing currently reflects the interests of a narrow
minority of people. A directed effort is needed to broaden the accessibility of computing and
amplify the influence of historically underrepresented people in shaping its future.
The tools for thought we want to build aren't just to help us do more thinking about our tools.
We're trying to make tools to help people solve real
problems in the world. That means we need to be able to talk about these problems. We also need
to be able to recognize and discuss problems within our own cultures, like the lack of diversity.
Those are just words. They are necessary, but not sufficient. I've received a number of suggestions
for concrete actions we can take as
a community, and that I can take as the community's steward, to make this a more welcoming and
inclusive place. I'm doing my best to put them all into effect, and am excited to share more of that
work with you when it is ready. Like accessibility, a focus on justice benefits not just the people
who are underrepresented,
but everyone.
So if you haven't yet reflected on how to make your own work embody the values of justice and inclusion, or moving power from those who have too much to those who don't, I implore
you to do that.
Finally, if there are things you would like to see me do, or changes you'd like to see
in our community to make it a more empowering space for black and brown people in particular,
I would be grateful to hear from you.
Welcome, Jennifer.
Thank you. Happy to be here.
Yeah, I'm very excited for this conversation. And in particular, I think a few of my longtime listeners are excited too,
because you have the honor of being requested.
Not all of my guests were requested, but you were.
Oh, that's always nice.
So I thought maybe we'd start with your background.
I know you've worked at a number of research labs that would be familiar to people in our community.
You've worked at the Media Lab, and it sounds like in two different groups at the Media Lab research labs that would be familiar to people in our community. You've
worked at the Media Lab, and it sounds like in two different groups at the Media Lab. Is that right?
Yes, Hilo Tech and Lifelong Kindergarten.
So yeah, I think most of us are pretty familiar with the Lifelong Kindergarten
because it's Mitch Resnick's group, and it's the group that made Scratch.
But I haven't heard of the Hilo Tech group. What's that?
Yeah, Hilo Tech is a group that was founded
by leah beakley um and leah is probably best known for uh being the creator of the lily pad arduino
oh yeah which is a version of the original arduino but fundamentally redesigned to support
e-tech style based electronics as opposed to more traditional electronics. So rather than
breadboarding, you connect to the lily pad with conductive thread and there's a whole set of
components that go with it that also are connected by sewing and conductive thread. The lily pad is
kind of something I knew about before I was interested in the media lab, but really a
seminal example of how redesigning an existing piece of technology
enables not just a different type of application for that technology, but also
really broadens participation to a totally different group of people. And in this case,
it was people who enjoy sewing or working with textiles. So Hilo Tech was a group that was
really founded around this notion of integrating emerging or highly technical platforms like electronics, computational design and coding with established or traditional or sometimes what are considered like low forms of making in terms of arts and crafts, fine art, traditional craftsmanship, but also arts and crafts in the sense that people
think of that as more of an approachable or hobbyist or accessible form of making. So the
group was all structured around different ways of supporting both communities and different types of
practices in combining these different modes of making and different materials. So there was a lot of work in the group being done
around combining electronics and paper craft, and this is being done by Jiqi, and she now has a
company called Chibitronics that produces these electronic stickers or paper craft and electronics
kits. Also Dave Mellis, who was one of the original Arduino team, did a lot of work in the
group to help people make their own consumer electronics. So radios, computer mice, speakers,
and then his most complex electronic product that you could build yourself is a DIY cell phone.
There was also a lot of work that extended the e-textiles platform or the lily pad platforms emily level in the group
did a lot of work around supporting community engagement and kind of creating a community
platform and sharing platform for the lily pad called lily pond and she also did some work in
building different versions of the lily pad that were simpler and easier to get started with
hannah perner wilson was also in the group. She's a rock star.
All these people are rock stars, but she really took the integration of craft and electronics
really far by combining her thesis was this work called A Kid of No Parts, which was basically
looking at all the ways you could repurpose materials that were not traditionally thought
of as electronic materials into platforms for building electronics and building sensors.
And so she did a lot of knitted resistive sensors that could be then integrated into
clothing.
She did work with conductive paints to create painted speakers on found materials.
She's also one of the people I respect the most in terms of dissemination and documentation,
where everything she did is very carefully documented and presented online. And if you're
all interested in craft-based electronics, Hannah is really the master in that area,
in addition to the work that Leah did. But Leah really brought this amazing group of people
together. And then I was lucky enough to be able to join it for my master's.
And that was really where I first got excited about this idea of combining code as a design tool and then as a design tool for making physical objects by integrating it with digital fabrication.
And so Leah was really the person who inspired me to start looking at the ways in which we can use computer programming to build really complex forms and objects and then
turn those things into real things by using 3D printing or laser cutting or computer-controlled
milling. So yeah, it's a quick overview of what Hyla Tech does, but it's no longer a research
group at the Media Lab, but we still maintain an active community independently, and we actually
have meetups biannually, and we're actually
having one this summer in Kentucky. So really excited about that. Oh, wow. And it's just for
people who are part of the group, or is it open to the public? No, it's still kind of an internal
element. So the original members of the group meet up and share our research and share our work.
We've kind of all spread out across the world at this point. I don't know what this says about the type of work that I'm interested in
doing, but often it's something that is at odds with a lot of like institutions. And so the
media lab itself was, I think there's a lot of really amazing things about the media lab.
There were aspects of high-low tech that weren't always understood or supported there. And I think there's a lot of really amazing things about the Media Lab. There were aspects of Hylotech that weren't always understood or supported there.
And I think, you know, the fact that Hylotech is not a research group there anymore is something that's acknowledged as a loss.
But it's also difficult to have non-traditional engineering communities in what are often pretty traditional or rigid engineering communities.
And I think high-low tech was an example of that.
So it still exists in terms of the work that we are doing as individuals
and Leah is doing, but is no longer a research group at MIT, unfortunately.
So I thought the name, I didn't quite get it until you were just explaining it now,
high-low tech.
There's this saying or this idea that high tech or technology is just things that didn't exist when you were born.
I think today's definition of high tech is things that didn't exist when you were born
that were likely made in California.
That's, I think, an approximate definition for what high tech is.
I think what's kind of interesting about the high-low tech group
is that it's kind of like trying to take things that were made after you were born, but make them
as usable things that you grew up with. Yeah, a bunch of it is thinking about ways in which we
can make different approaches to making in different media and material for making more
approachable or more inviting or more relevant to different
groups of people. And often that's groups of people that traditionally have been excluded
or viewed as not relevant to them. And so in some ways that's lowering barriers to entry and making
things easier. Although, as I know, you're very aware of and your audience is very aware of what
it means to make something easier is not a simple notion. There's a lot to unpack there. But it's also often just this notion of, I would say, what are the
expressive opportunities that come with reframing a form of making along the traditions of another
form of making? So in traditional circuit design, the trace is primarily viewed through a notion and like the trace on the circuit board.
Be careful. Trace can mean something different in programming.
The actual circuits themselves are primarily viewed purely from a functional and efficiency perspective.
So is it thick enough to support the necessary electrical properties and does it connect the necessary components?
But if you translate that
to the trace is also something that you're embroidering with conductive thread or you're
drawing with conductive ink then it's both a means to transmit electricity and have a functioning
circuit but it also becomes a decorative or an aesthetic element and that's something that comes
out of traditions of drawing and embroidery and sewing, where functional aspects are also things that contribute to the visual properties,
the beauty of a piece. And that's something that I think everyone in high-low tech felt was very
powerful and very meaningful. So it's an opportunity to engage people who maybe have an interest in drawing or have an interest in sewing or in craft in seeing themselves as producers of technology, which is very important.
But it's also an opportunity to re-envision what technology might look like or what these materials might mean and how they might be used to people who kind of have a closed notion of what a circuit is or what applications
computer programming is used for. And so one of the really interesting experiences I had in the
group is when I was first doing computational design workshops with processing and I was having
people write code to make designs for lamps, for example. I would work with people who had never
programmed before and this was their first introduction to programming and they would
be interested in that. But I'd also work with people who had never programmed before, and this was their first introduction to programming, and they would be interested in that.
But I'd also work with experienced programmers and software developers who would say, oh, I had no idea that code could be used to make beautiful forms.
I feel like both of those are powerful outcomes.
So I find myself wondering how you identify, and I have these two myths in my head that I think could apply.
One is the myth of the guy who brought fire from the gods.
Oh, Prometheus.
Prometheus, yes.
So you could see yourself as Prometheus as being like a technologist who's bringing technology to artists.
Or you could see yourself as more like Jack in Jack and the Beanstalk,
like someone who goes up to the gods and steals their stuff and then comes back down.
So do you identify more as a technologist who's democratizing technology
or as an artist who's trying to claim technology for themselves?
Yeah.
I would say there are elements of both of those that are things
that I think are important and interesting and supporting,
but I wouldn't say I view myself precisely through either of those lenses. So one
thing that I continually struggle with is the language that I use to describe what I'm doing
and who I'm doing it for. And a lot of this comes up in talking about artists versus programmers.
The reality is that while not all artists program, it's also not an accurate distinction to say
that there are programmers and then there are artists.
Much of the work I've done has been inspired by artists
who are some of the best programmers
that I also have ever met.
And also some of the best software developers
and application developers that I've ever met.
So I would see it more as trying to explain to artists why programming is so great
or help make it easier for artists to program because artists do very difficult things all the
time and learn very difficult things all the time. But more thinking from the perspective of
what are domains of art that could really benefit from some of the powerful affordances of
programming? Because I believe that, frankly, at this point, everything can benefit from programming
if it's presented in the right context,
but that we might rethink
how we develop programming environments
or programming languages
to support these specific domains.
And that's less about, I would say,
just making it easier for some artists to use programming
and more about thinking a little bit
about changing the idea of what programming might be or changing the notion of how it might connect
to different forms of expression. And at the end of the day, this comes down to my own experience
and the things that motivated me personally. I often talk informally about my work
as being a series of productive failures
and trying to mash symbolic programming
and drawing together.
And because there are these two forms of creation
that I feel are incredibly powerful personally,
but have fundamentally different properties
in many respects.
And I can talk more about what those are at length. But a lot of the work that I've done
has been trying to approach different ways of how can we get these two things to be integrated more
closely together? How can we make the experience of programming more like drawing? Or how can we
make the process of programming reflect some of the key forms of
expression of drawing and that's translated out into a bunch of different domains beyond drawing
I think it's more of a notion of taking these two forms of expression that are really powerful and
thinking about different representations different interfaces even different forms of teaching and
facilitation that allow us to combine them as opposed to
introducing this audience like artists who are previously unaware of this medium programming
to it and helping them to get started that's definitely a part of it but not the whole story
from the other perspective I think you use the metaphor of jack and the Beanstalk. I think there's another angle of this work or a question that
I ask myself, which is what if at all value does my work provide for more established programming
communities like software development? And that's only been a question that I've been focusing on
relatively recently. And I think it's not the primary focus of my work, but there are a lot of things that I've been looking at in terms of how
artists program or how artists work, and this is generalizing to some extent, but often true,
but they often engage in an exploratory working practice where they don't start with a clearly
defined problem and say, okay, now I just need to break this into smaller problems and solve it.
They have a process where they're open to mistakes, they're open to variation, and they kind of explore as
they make. Victor has talked about this in a lot of his writing as well. That's something that's
true for artists and often goes against very traditional notions of what good programming
practice should be or what good software development should be.
But I also think that software developers work in an exploratory manner.
And other people have written about this in many cases. And I think that it's impossible to predict and plan every aspect of a programming project before you execute on it.
There are always surprises and unexpected things and experimentation.
So I think there are things we can learn from how more messy forms of creation or more messy creative practitioners use programming that might help us make better programming tools for people
like software developers or other communities. But I haven't explored that direction in my work
as much. Just to clarify, when I was talking about
the Jack and the Beanstalk metaphor, the metaphor was artists are more normal people and then the
giants are the programmers. And so you're like stealing some of the programmer magic and bring
it down to the artists. Yeah. Yeah. I'm sorry. I kind of... You flipped the metaphor as like
artists where the people... Because you could do it either way. Yeah. Yeah.
I mean, there is so much magic in programming, but I don't.
I guess just to give you examples of people that come to mind, like Seymour Papert to me seems like someone who was like really a mathematician who was trying to like bring mathematics to people who didn't like mathematics.
As opposed to some like an artist who wanted to use mathematics in their work.
He was very much, I felt, like a democratizer
when there are other people who are like scrappy artists
who just take whatever they can and embed it in their work.
I mean, I do think there's this problematic notion,
and I've heard this addressed on your podcast before,
but this problematic notion of certain
types of people being equipped to be good programmers and other types of people being
less equipped or a notion of exceptionalism in certain forms of programming. That's unfortunate
and not surprising, but I do like to try to break down that idea a little bit that while programming is
an incredibly powerful form of expression, while abstraction has value in so many different forms
of design and so many different forms of expression, while symbolic mathematics has value
in so many different forms of expression, programming is too vast to say that there is one personality type or one intelligence
type that really aligns with it. And if you don't have that, you're never going to truly
be effective in it. That's definitely a belief I had about myself getting started in programming,
and it was totally by accident that I got started programming. But I see it now with some of the
people that I work with who,
for example, a lot of artists who use visual programming languages like Grasshopper,
which is the node-based data flow language for Rhino, which is a very powerful programming language. And I'll have people tell me, oh, I don't know how to program. I just use Grasshopper.
It's like, why is that not programming? And where did you get that idea? Because
it is a visual programming
language you know and there are people but i think fewer and fewer who don't consider visual
programming programming but um definitely not uh from where i'm sitting fewer and fewer right right
right okay well i've had the good fortune of being in a very supportive environment in lifelong
kindergarten around this i guess and i'm still learning a bit about how to rest the world. Yeah. You're like in Mecca. You're like,
oh, everyone believes in the same God as me. Everything's wonderful.
But so I do think it's important to dispel those ideas and dispel notions of also like
where programming is relevant. I don't know how much oxygen I even want to give to that space,
but needless to say,
I focus more on not so much,
you know, is this programming
or is this not programming?
And more, what are different people trying to make?
How can the ideas or the approaches
embedded in programming support them?
What are the barriers?
And how can we help more people make things
by building different interfaces or environments or tools that get around some of the barriers? And how can we help more people make things by building different interfaces or environments or tools
that get around some of those barriers?
And there's so many questions and challenges in there
that at the end of the day,
there's somebody who looks at what I do and say,
yeah, but you don't have conditionals in this language.
So it's not really a programming language.
I'm like, well, there will be other issues
where we need conditionals,
but I want that to be driven by the people I'm trying to support
or the work that I'm trying to support,
not by the fact that there's a notion that
to be real programming, you need conditionals.
I feel like I want to just keep talking about this sort of stuff,
but I think it probably might help our audience
to talk more concretely about some work you've done.
Yeah, absolutely.
Actually, before we do that, I want to just ask you a question, which I might end up cutting, but I'm curious to know the answer.
Sure.
I really enjoyed the who's who of Hilo Tech and all the interesting work happening there.
And I was wondering if you would do the same for the Lifelong Kindergarten group.
I can tell you who was present when I was in Lifelong Kindergarten because it's since grown. So the group is led by Mitch Resnick, who I am sure many
people are familiar with, but he was a student of Seymour Papert and one of the main designers and
really the continued architect of the Scratch programming language. Although Scratch is really
a community effort, It's come together
from the contributions of a lot of different people. While I was there, there were a few other
grad students that really influenced me. So, Rikaros Roke, who is now faculty at CU Boulder
in the information science department, I think. She was doing a lot of work around facilitation
in computational learning,
specifically related to families.
So how do you help not just kids learn to program,
but how do you engage people in computational learning
in the context of a family?
And the cool thing about that work
is that she is an immigrant,
comes from an immigrant family,
was focusing on working with other immigrant families by and large as a means to think about what are some of the constraints that different communities people have when engaging in different types of learning.
And how can we have this be a process not just to help people develop new skills, but also something that engages parents and their children together
in working with a new technology.
Rick Rose's work really stands out to me because facilitation is such an important aspect of
how we think about people learning programming.
Often the work in my domain that gets a lot of focus in research is what can an environment
do?
What can a technology do to make learning easier? But there's also so
many important questions about what it looks like to facilitate a learning environment,
what the interactions look like between the people who are quote-unquote experts and the
people who are non-experts. And Rick Rose is one of the people who's really doing deep work in that
space. Simon Dudescupta was another grad student who was in the group when I was there.
He is doing really interesting work of combining data science and programming for young people,
just like Seymour Papert and Mitch said that kids are programmers and we should build programming languages for them.
Saimindu has kind of extended that to say that kids can be data scientists.
And he's looked at ways to use Scratch as a way to help kids do programming with data around their own data.
So the Scratch community has a lot of opportunities in that space where kids can get data about how
many, you know, the way people have viewed their projects, the people they've worked with in the
community. And so he's done a lot of really interesting research
in that area.
And I think that there are some scratch extensions
that have come out of it
that specifically are geared towards programming
with external data.
So, oh man, there were a lot of other people
in the group when I was there.
Eric Rosenbaum was there.
We overlapped.
He and Jay Silver did the Makey Makey.
And Eric has done a lot of really great work around computational music creation that's continuing to be embedded
in Scratch. One of the great things is that a lot of students in the group are also members of the
Scratch team and will come back in various ways and continue to contribute to Scratch. I think
Eric is actually now working as a member of the Scratch team. Natalie Rusk was there as a research scientist.
I believe she's still there. Yeah, I've heard her name before. Yeah, Natalie is another big
component of Scratch and also has done really thoughtful work in the space of motivation
and understanding what drives people to engage in difficult learning tasks
or learning challenges. Then there's the whole Scratch team, which one of the things I feel
really fortunate about being in lifelong kindergarten was not just interacting with
the grad students, but the fact that there's this basically small company of people who are
maintaining and designing and doing the engineering for the
Scratch community and Scratch programming environment. And they're such a wonderful
group of people, but also have such a wealth of knowledge around design, around working with
people, around teaching. And a lot of the work that I've done was directly influenced by talking
with those people about how they thought about designing visual programming languages or how they thought about developing documentation. And I think it's
something that is maybe often not present in academia where you're maybe working in a simplified
or more less ecologically valid context and less aware of what people in the quote-unquote real
world are doing. There's too many people in lifelong kindergarten to talk about, but those are a few people that really influenced me. I think for people who
aren't in academia, academia sometimes seems overwhelmingly large. But when you meet a few
people who are in academia and you just like ask like, who are your friends? Like, who are the
people that you've worked with and are interesting? And they just like list 10 people and you're like,
oh, I can learn about those 10 people. Yeah. Yeah. There's too many people to talk about. But the other nice thing about both Hilotech and Lifelong Kindergarten is
everyone I've encountered in those groups has been a very kind person as well. And so if people
are interested and they look into, you know, Simon do a Rick Rose's work, I encourage them to reach
out because they're all very kind and interested in sharing and connecting with people. Cool. Okay, so now let's get into some of that more concrete stuff of the work you've done.
There was one talk of yours I saw, you had two different projects.
One project, you did some testing and evaluation, then you did a second project.
So you're probably thinking of Para and Dynamic Brushes, I'm guessing?
Yes.
Got it. I forgot the name Para. The second one and Dynamic Brushes, I'm guessing? Got it. I forgot the name Para.
The second one, Dynamic Brushes, I'm familiar with.
Yeah, it's not a great name, but we needed a name.
Well, I like Dynamic Brushes a lot, actually.
I have trouble with the phrase procedural art
because procedures are just one way of describing things. Yeah, trouble with it too but i haven't found a better word so i guess like i like maybe
dynamic art or computational art you know something more abstract than procedural because you know
yeah yeah um i i'm continuing to refine the language I use. I chose procedural because it's one that
both in the art community is used and also historically with regards to art made with
computers and also to some extent art that is not made with a digital computer, but arguably
is computational, like instructional art for example um solo wit
is kind of the canonical example in that space that's interesting that there's this connection
to instructional art even before computers or like giving like a recipe of instructions even
before computers were around yeah well we can go way back into the history of all kind like
all kinds of forms of algorithmic production that existed before
computers yeah interesting yeah yeah like you could yeah i can imagine yeah like uh an architect
could like you know explain to the brick layers where they want the bricks to go yeah and i'm sure
you've are familiar with christopher alexander's book a pattern language and all the work he's done
that's all about arguably abstraction in in, in architecture, in structure. But knitting, weaving are kind of the canonical
examples of algorithmic forms of making. It's interesting because, not to go on too much of
a tangent, but it's hard not to. A lot of artists think about what they do
algorithmically, even if they don't program, because artists are highly reflective in their
process. They're very interested in what they make, but they're also very interested in how
they make it. So for example, Chantel Martin, who's an artist that I'm very inspired by and
had a lot of conversations with her about her process and drawing. She creates these giant like large-scale drawings all improvisationally and all
seemingly without a plan but she talks about the fact that oh yeah I understand this general
structure of when I make a decision to vary the line, how long that I draw, the general distance
between different elements of my drawing, you know, when I choose to
create these different singularities or inner drawing, like exceptions, and she doesn't program,
or she hadn't really when we were first talking, but she was thinking about her work very procedurally,
and she was then very drawn to the notion of programming, and since gone in to do work in
this space, the idea not just that it
would let her create different things but that oh this might help me see the process that I use
through a different lens through a different representation is she one of the artists that
you commissioned to use your tools to try it out not yet hopefully soon she's she's uh unsurprisingly really blown up recently.
Not recently, but yeah,
this is the problem of talking with really talented people
is they're very successful
and then they have a lot of projects to do.
We've talked about doing projects together
and hopefully eventually that will happen,
but she's also doing a lot of work on her own, right?
Yeah, she's very successful at this point in time,
which is great.
So I liked in your presentation about these two projects
how you framed it that you were doing
interplay of the dictionary systems engineering.
There was this cool graphic
that you started from certain motivations.
Anyways, so maybe you could walk us through the process.
It builds very much on the notion of user-centered design or other human-centered forms of design practice, where it is to start with an understanding of an existing domain and examining what people are doing in that domain.
What are the opportunities and what
are the challenges, and then starting to try and build tools and technologies that try to address
some of those challenges. But the idea being that you're really starting first by trying to
understand the practices of the people you're trying to support. And so in the case of artists,
like I do a lot of work initially with observing artists at work in their studios, interviewing them, analyzing their work and
kind of making hypotheses about how they might be doing certain things and then asking them about
this and talking with them about it. And then trying to prototype tools that might align with
some of the things they're trying to do. To start with Para, one of the things that I noticed really early on when doing workshops with people with processing is that often people
were coming from a background of using direct manipulation tools like Illustrator, and
processing generates vector graphics, so it's in many ways really powerful to combine it with a
tool like Illustrator. But there was this big disconnect, both moving between these different
interfaces, symbolic textual programming and direct manipulation of vector graphics, but also
these very different representations where you're using this imperative representation and processing
where explicitly every transformation you're describing, whereas in a direct manipulation
tool, it doesn't matter how many times you move something.
All that matters is what the final position of that object is, for example. And so kind of seeing
these tensions come out and thinking about and then informing them through conversations with
people about why they choose to use an imperative programming tool like processing or why an artist
might choose instead to use a tool like Illustrator. And that
really led into some of the early prototypes for Para, which is this direct manipulation procedural
art tool. And so I don't call it a programming tool because there's no symbolic code. There's no
code that's displayed to the person using the system. But there are types of things that you can do with it,
like describe constraints,
have iterative variation between multiple objects
that normally you would use a programming language
to achieve these same relationships.
And the development of Para,
going back to this interdisciplinary systems engineering idea,
the challenge in building the system is,
as you're building it, you're making all the,
everyone who's worked in software development knows this,
you're making all these individual decisions at any point
that could be really impactful
or are just necessary to get the thing up and running.
And the task is to understand at some point
how to inform those decisions.
And so the approach that we've taken
is to actually do iterative testing
of the system with professional artists using it. So as soon as the system is in any shape for
anybody to use it, which is often like way too early for my comfort, but probably too late in
some ways, I should probably continue to try to do it earlier. We'll have professional artists use it and try to make work with it and observe them using it
and actively modify it in response to some of the challenges that they observe.
With Para, we had a professional painter use the system for two weeks while we
looked at what she was doing. This is the painter Kim Smith, who's also at the Media Lab. And then
revised elements of both the interface and also some aspects of the underlying representation
in response to things that just didn't make any sense. So in some ways, the ways we were
representing lists were really powerful from an abstract notion in that you create a list and
then really it was very was very simple it was
just a an ordered collection of vector graphics but not very useful from an artistic standpoint
if you create this collection and you have no immediate functionality that comes out of it so
we we added the ability when you create a list that it is maybe getting too deep in the weeds
but that it automatically had some constraints on those objects imposed so that when you would
start to manipulate them
through direct manipulation,
you would get useful changes
across the collection of the entire objects.
This is something that came directly
out of working with this one artist
and in the process of watching what she did with it.
And that process is something that improves the tool itself
and also helps us to understand more general notions
of like,
how does it actually apply to longer term use? Does it support the creation of different types
of art as well as supporting kind of these interesting small interactions? At what point
does it stop becoming interesting for the artist? And how do we need to think about different forms
of expression that we should enable? And then also, really, what are the limitations of the tool?
So really, towards the end of the second week, Kim started using Para to make more illustrative work in the system
as opposed to abstract forms.
So she was using the ability to automatically manipulate large collections of objects,
to create patterning in an illustration of an animal and some plants. That was really
interesting to see and was really exciting, but we also noticed right away that compared to some
of the illustrations she had done, you know, with watercolor by hand, the fact that she was using
vector graphics was a kind of a big constraint in terms of those forms of expression. And so
that's where this like cycle kind of feeds back
into building into the next project, where you look at what people have done, it helps you
improve the existing system, but it also points to these directions to new work and new systems.
And so without doing the studies with Para, with artists, we wouldn't have thought more about,
well, should we examine these assumptions we've made around vector graphics
and direct manipulation of vector graphics as a way of supporting more expressive forms of
procedural art for artists who work by hand and focus more on this idea of drawing and actually
drawing with a pencil or drawing with a stylus. And that's where dynamic brushes came from.
And that's not to say that I think that Para was a dead end. In fact,
it's something that we're still working on. And that project was a collaboration with Joel Brandt and Radimir Mech and also Sumit Goja. But Joel and Radimir are both research scientists at Adobe.
And, you know, there are ideas from Para that are continuing to percolate through through adobe in different forms but it also
sort of opened up new new ideas or or different directions that we might go and and that's kind
of where dynamic brushes came from yeah before we continue on in the story i wanted to just
talk a bit about um para i think just just to give people a mental picture if something didn't
come to mind uh i think the tool looks a bit like photoshop's like a normal graphics editor it
actually looks more like illustrator but yeah it's similar yeah it has a toolbar it has a what
appears to be a layers panel and then a canvas and a few other things yeah yeah yeah that's that's
yeah i just want to give people a bit of a mental image and then the way i I thought of it is, like you said, it's not a programming language.
It's first and foremost a tool like that, but then it seems kind of augmented with these cool computational constraints and linkages,
which I've seen actually a few tools start to incorporate.
For example, Figma has this new components, like master and instance ones,
that you can change the master and all the instances will change too so it seems like this tool was exploring that space yeah i should say um there's
been work in the past that has some of these same elements and a major frustration i have in in
research in general is that i feel like we're often discouraged from building on prior strong
work as opposed to um distinguishing our work as completely novel.
But Para has new things in it, and I would say represents a different direction than some
previous work. But everything going back from Sketchpad to Morphic to the work that Toby
Shackman has done in Apparatus to the work that Rubiat Kazi has done with Kitty,
and then also obviously all of Brett's work in dynamic data visualizations. It's definitely
built on a strong foundation of prior works. I would say that the things that maybe don't get
focused on in Para as much that are important to me are how we tried to position those things with respect to the specific domain of
visual art and illustration and then also really how we evaluated it with artists in terms of
looking at the types of work that it could enable and the ability to support both more abstract
forms of procedural art but also these more illustrative forms with animals and
plants and so on. But yeah, it's built on this foundation and inspired by and hopefully contributes
to this larger body of work that I think is really important. Oh, also sketch and sketch is an
important example in that space too. But yeah, this body of work that's looking at what are the
ways in which we can express procedural relationships, programmatic relationships
through direct manipulation. And there's so much exciting work that's been done in that space and
so many difficult challenges to continue tackling. So the list improvement you made that you went
over, I thought that sounded interesting. It sounded like before when someone would make a
list, it would just be a collection of items, but they wouldn't be linked at all.
And then so you changed it so when you put things in the list, they are automatically constrained to each other in some way?
Yeah, I mean, one of the first ideas we had was really just what are powerful ideas from more conventional forms of programming that we can apply to the direct manipulation context and have them so represented.
And the notion of a list generally is just something that can serve many applications,
but in itself is specifying an order
and a collection of data,
but not specifying operations on that data necessarily.
And although that is useful
to have sort of bare bones lists off the bat,
as we quickly observed
when you're coming from the space of direct manipulation and also maybe not thinking about collections of artwork in that way.
What's really powerful about Para is that you can define a relationship and then start to play with
it by manipulating the shapes themselves, by manipulating the actual artwork. And the powerful aspect of direct manipulation in general is that immediate feedback where you
change something and immediately get a sense of how it changes everything else.
And so to have built-in constraints when you're applying lists, so that when I apply a list to
10 objects and then drag the last object and it redistributes the objects gives people immediately a sense of some of the potentials of using those types of relationships as opposed to they describe a list
and really the structure is there but but nothing happens immediately there was a whole bunch of
additional work in general we did in para of thinking about oh what are these really complex
types of procedural relationships you might create. So we had something that looked
a little bit like prototype inheritance in the system, where you would have something that when
you duplicated an object, it would maintain the same properties as its parent until you explicitly
manipulated one of those properties, and then it would have its own value for that. And that was
very powerful, but you really had to have not only a deep understanding
of where you might want to use that, but also it really upped the burden of how much complexity
you had to manage in the system and how we might even represent those relationships or represent
that complexity. And so in the end, we settled on this simple set of procedural concepts, constraints, lists, and declarative duplication
because it kind of hit this sweet spot of letting you do things you couldn't do in existing direct manipulation tools,
but not increasing the complexity so much that we really had to, you know,
like it's very rare that you would encounter a situation where you're creating a cyclic constraint in para for example you can and the system will alert you if you're doing that
but because you could do so much just with simple object to object constraints or do a single
constraint and map it from one list to another list a lot of that complexity was unnecessary
i would say going forward one of the biggest challenges we have in thinking about direct manipulation as a format for expressing programmatic-like or procedural relationships is how do we help people debug these things? if you can't unpack them, if you can't evaluate them when they're functioning incorrectly, then it's not going to be any easier than other forms of programming. And to be fair,
this is already an issue we have with regular direct manipulation systems where they don't
necessarily always perform super well when you have really complicated compositions or large
collections of artwork. Like the layers panel is not the greatest panel for manipulating an artwork with assets
on the order of the thousands or even the hundreds.
And there's people doing work to address this.
But I would say like if and when I continue to do work in the space of direct manipulation
based procedural or computational art tools, debugging and
understanding structure is really an area of interest for me. So one of the questions that
I still have about this list improvement you made, did you simplify lists from this generic
structure to have default behavior, to have standard behavior, or was it just that you
gave lists a certain default that could be undone if they wanted it to be?
No, we just gave them a default.
You can always remove in Para.
You can remove any constraint that's defined.
We didn't actually take away any functionality.
We more chose what, for the context of this particular study, was an effective default.
I remember, yeah, learnable programming does a great job of explaining why good defaults are just like so important, but there's such a simple way to improve the learnability of a tool.
Yeah. I would say this is something that often gets not as much, I'm glad it was highlighted
in that essay. I don't think it gets enough appreciation in general, like something that's
often lost on people who don't understand the power of Scratch is not understanding some ideas of really powerful
but simple starting points they provided people,
how they introduce different concepts.
Microworlds is kind of a really important idea in this space.
But then I would say that I think there are a few examples
of places where Scratch goes kind of too far
and they don't let you unpack the glide like the glide command i have a lot of
trouble with because interesting and i i like you know tell my told my students you know to never
use it to you know and it's harder initially because they have to learn uh they have to like
point their object in the way they want it to go and then move in a loop so it's like three blocks
instead of one but it's so much more powerful to understand the primitives in that
way than it is to glide is is so limited um and so yeah anyways yeah no it's that's interesting
to hear i mean i think i think this is the general tension you get when you have a pretty
diverse audience of people using a given tool. I know one of the
challenges that the Scratch team is always dealing with is how can we support people who are doing
really complex work in the system without making it so that when completely new people encounter
it, they might be encountering types of code or types of blocks that make it really difficult for
them to do simple things. And there's always a push and pull in terms of how you resolve those
challenges. My interest as a researcher working in the space of designing creative systems is
finding good resolutions to some of those challenges, but moreover, trying to do a better
job of documenting and explaining
how those decisions were made and why they were made. I feel like this is a struggle in writing
about any software-based project is there's a lot of design knowledge and information that gets
kind of glossed over because the process of building a software tool is made
of so many important small design decisions that really converge to make the tool what it is but
it's very difficult to communicate those decisions in a short and understandable manner to an outside
audience there are other things that happen in para that i haven't been able to unpack and talk
about um and we're definitely returning to the system to continue to work on it in other areas.
But yeah, this is an ongoing challenge in the work that I do.
Let's talk about dynamic brushes now.
And maybe you could pick up the story where you left off how the evaluation and feedback you got from Para led to the conception of dynamic brushes.
Yeah, so it was sort of a
convergence of factors one was really getting to the limitations of vector graphics where they're
very useful for some forms of expression but if you're someone who draws or paints
and many of the people that i was working with fit that description. There is so much
expressive power you get just from the movement of different people's hands. Just any anybody
draws differently than anybody else no matter their level of experience. If you look at people
who have a lot of experience in a form of manual draftsmanship let's say like painting or drawing, they have spent years
refining how they move their hand to draw. I spoke at length with my former undergrad advisor,
Michael Salter, who's a illustrator and studio artist. And he talked about how the opportunity
of drawing is that the fastest way for him to express an idea in his head was the length of his arm and that he
really refined this ability to draw and express you know ideas that came into his head very quickly
and so wait sorry uh what about the length of his arm oh it's sort of a i should find the exact
quote um but but that the the you know the fastest transmission of an idea from his head to reality was basically this mapping between his head to the extension of his arm.
I'm misquoting him slightly there.
So apologies, Michael.
Yeah.
We're like someone who's good at talking.
It might be the shortest path is from their head to their tongue.
Yeah.
Yeah.
Maybe.
Yeah.
Yeah. Yeah, yeah, maybe. Yeah, yeah. It's this common notion that has come up, the speed of drawing
being something that's both powerful in terms of getting ideas out there, but also the thing that
affects what it looks like. So Chantel talks about how she tries to draw without thinking very
quickly, because if she thinks about it too much, it'll change her style. And it's almost like it
has elements of like meditation in it i think to some
extent i think seymour pap would really like this kind of conversation because it's all somatic
somatic knowledge body knowledge yeah yeah uh yeah there's definitely uh this notion of yeah
embodied or tacit knowledge i think uh that idea was really what led to dynamic brushes of thinking,
okay, let's take the same approach we took with Para, but reframe it around this notion of
drawing by hand. So you're not using vector graphics, but what you're doing is drawing with
a pencil, or in this case, like a digital stylus. And how can we use that as a starting point?
There's any number of directions
that might have taken but then I was also having a lot of conversations with artists who regularly
used programming in their work and I focused on talking with artists who also did these works that
had what to me seemed like very hand-drawn qualities to them. So Eric Natsky and Emily Gobiel and Theo Watson.
There is a common pattern there where they were building basically their own software
that let them draw, but then procedurally manipulated aspects of their drawings.
They were drawing with a stylus, and then Eric wrote software where
it would take the stylus input and manipulate the line. This is simplifying it, but manipulate the
line by sampling from some other external data like a JPEG image. If you look at his work,
some of his work from this time, it has these really sweeping forms, but then this really
complex variation on the weight of the stroke and the color of the stroke that's coming from this functionality of software he wrote.
And the same thing with Emily and Theo had a really interesting process where Emily comes from both a background in programming, but also in illustration and puppet design.
And Theo comes from a background that's more rooted in software development and computer science. And they have this wonderful collaborative process where Theo
would kind of write these ad hoc scripts in Open Frameworks, which is a tool he was one of the
co-founders of, where it would do something really simple but really playful, like take
the line that Emily was drawing and then also take microphone input and modulate some aspects of the line geometry based on the
amplitude of the mic input for example or take some form she had drawn and then automatically
do a circle packing algorithm within it and fill it with with circles and Emily would use these
and then generate ideas of oh what if we did instead? And Theo would script up another tool as a variation, and they kind of work back and forth
and build these ad hoc drawing tools. It's like, oh, that's amazing. That's such a powerful process.
Can we think about the idea of a programming language for drawing as something that is
actually a programming language for building your own drawing tools that take your drawing input and transform it in some way that the artist dictates or describes.
And that's really where Dynamic Brushes, that was kind of the start of that project, is that idea.
I just have to say, you're a true computer scientist. You're like,
you're starting with drawings. You're like, oh, well, this is nice. But like, how do I
make these drawings a little bit more dynamic? And you're like, okay, I'm going to make a tool for artists to make dynamic drawings.
Okay, that's a good idea.
Oh, wait, you know what?
I need to make a tool that artists can use to make their own tools.
Yeah, the next project is going to be,
you're going to make a generic programming language
that will allow people to make tools that will allow other people.
Yeah, it's just, you're never going to stop.
Yeah, no, it's turtles all the way down.
Yeah, I think it's a natural process when your goal is to think about
how can I help people be expressive with programming?
That's basically what programming enables you to do
is build your own,
tools is not always the right term,
but your own subroutines, forms of automation.
Often artists I've talked to who use programming extensively describe it as programming and it lets you build a medium
and so it's useful to take other aspects of programming like oh can we just help people
automate this one thing that everyone needs to do that's's powerful and that's valuable. But I think the more I think about helping different people be expressive, you really want to give them the full access to the
true power of programming, which is constructing your own software, basically. I like the way
you're doing it in almost like this DSL kind of way. Yeah. It reminds me of like the LMK Steps
project where you have like a main interface that's easier to use
and you have a specialized interface to just specialize the first interface
and then maybe an interface to specialize the specializer.
Yeah, there's all kinds of problems that come out of that
and I'm happy to talk about those at length.
But I think in general, I'm a big proponent of this notion
of domain-specific languages and dynamic brushes. Although it
brought a lot of challenges, I think the idea of, okay, let's build a language around this notion of
managing drawing data with a visual programming language and then letting people work with those
tools in a more traditional direct manipulation drawing environment. So the data from your stylus is now going to be represented
as absolute position over time, or its relative position over time,
or the change in force over time.
And once again, this was a collaboration with Joel and Radomir,
and then also Mitch was hugely influential in this work.
We borrowed a lot from work in the space of computational music
manipulation, where they're using very simple waveforms to modulate sound or generate sound.
And we then just simply applied them to modulating the line that was being drawn,
the position of it, the geometry of it, also the stylistic properties of it. So you can have a brush in dynamic brushes that follows the stylus
X and Y position, but modulates the geometric scale relative to the origin along a square wave.
So you get this geometric variation, but then the position is totally determined by how the
different person is drawing and so it's
this incredibly simple program from one viewpoint the program itself is very short it's not always
a measure of simplicity in programming but um but then you give it to a different artist and
each person makes something different with it whereas in para although i liked para a lot and
i still love it a lot a simple program because it was using the same geometric primitives,
there was a much higher threshold for different people generating different things off the bat.
And so that's probably the most powerful thing about dynamic brushes is that it just brings in
this form of data that's highly expressive and you can do a lot with simple programs. You can also do
very complex things. And so one of the key goals of the system was, let's not just see what
happens when we treat drawing as an input and let people manipulate it, but let's enable people to
create tools they couldn't make with other, that aren't available in other drawing software, like
Illustrator or Photoshop, where they can have things drawn that correspond with what they're
drawing by hand, but they can also have things being drawn automatically or independent of what they're drawing by hand.
And so I don't know how in detail you want me to go into the programming representation.
I do want you to go into it because I think it's fascinating.
I just want to draw a mental picture and I'll do a quick sketch and you can fill in some of the details.
Yeah, yeah.
So dynamic brushes, the pictures I saw was you have iPad and a computer screen.
On the computer screen, it looked a bit like scratch.
You had blocks that you could connect.
Then on the iPad, you were drawing.
The thing you were coding on the desktop screen was the brush.
As you changed the brush on the desktop and then you used your iPad pencil on the iPad,
different things like squares would come out of your pencil instead of just a line.
Yeah, yeah, exactly.
So the basic idea was the desktop environment had a visual programming environment in it
that gets compared both to Scratch a lot, and it probably owes more to Scratch than anything else,
but also gets compared a lot to node-based data flow programming like Max, MSP, or Grasshopper. And I would say it's actually what you're
creating are state diagrams. So it's a state and constraints model, which... State charts?
Yeah, yeah. We looked a lot at Steve Oney's Interstate as a way of informing the design of
this tool. So it's technically not the same
thing as the way the Dataflow languages in Grasshopper work, but it looks similar in that
transitions are event-driven. It's similar in some ways to Macs in that regard. Then the iPad
environment looks kind of like a really simple version of a tool like Procreate or some other tablet-based bitmap drawing environment.
But instead of having like a tool palette that is defined up front,
the tool palette is whatever programs that you create
in the programming environment, and they're linked.
And you said bitmap. We're not doing vector graphics anymore.
No, and the reason we did bitmap was really because
we started earlier with dynamic
brushes, testing it with artists. And we were working with people who had worked primarily in
a digital painting scenario and they were used to bitmap tools. So they wanted opacity. They wanted
the ability to feather the edges. You can do that with vector graphics. And there's a version of
dynamic brushes that is completely vector, but we haven't returned to that but it was more just the audience that we were looking at there's no reason it couldn't be a
vector tool necessarily but we chose to align it with bitmap drawing because that corresponded with
the types of people that started using the system early on cool so yeah i think i interrupted you
you were going to talk about
some of the programming model stuff,
the interesting bits.
Yeah, yeah.
It's so funny.
I usually give this with some images behind me.
So it's a useful practice to describe it without images.
So the kind of core,
I find I, I guess I often design my languages
around three primary programming concepts or components. And so in Para, it was
constraints, lists, and declarative duplication. And in Dynamics...
Why three?
Ah, it's like the magic number. I don't know. I guess, arguably, in Dynamic Brushes, you could
say it's four. But it's primarily data bindings, so just declarative mappings between some input
data, most commonly the stylus, but then you also have the synthesized data, like function generators, like a sine wave.
And then you also can import in external data.
We also added the ability to use mic input, the accelerometer of the tablet. can be mapped to a brush property. And the properties are obvious things like position and color and opacity,
but then also less obvious,
but very powerful things
like geometric transformation.
So rotation, scaling,
and then you describe data bindings
inside of states.
So that's the second thing.
And then this is basically
whenever a state is active,
the data bindings within it are active.
And you can only have one state active at a time, just, you know, traditional notion of state
machines. And transitions between states are driven by these event-driven transitions that
we predefined in the first version. So you have a limited set of events, kind of similar to Scratch,
like stylus events, when the stylus is down, when the stylus is up, or when a certain interval of time has elapsed, or when a certain interval of distance has been traversed.
And then we're working on a new version, which has a more expressive event definition structure, so you can describe more arbitrary events or evaluate arbitrary conditions. Yeah, so states, bindings,
transitions, and I guess the fourth thing would be then there is also a small set of actions, which
are just very simple methods that are called once, and they can be called once on any transition that
occurs. For example, you can initialize a new stroke when a transition occurs, or you can start a timer, which is really
useful for doing time-driven transitions. Or you can also call this action called spawn,
which is the most powerful, but maybe most complex aspect of the system, which basically lets one
brush generate another brush or multiple instances of another brush and so the reason we added spawning
to the system was because of what i was talking about several minutes ago where we wanted the
ability for the artist to create brushes where multiple things were happening simultaneously
you had one input and multiple outputs or you were drawing on the canvas but had some automated
things that would be drawn in response to what you were drawing.
And so spawning basically lets one brush generate copies of itself or brushes generate automatically that draw on a timer interval as opposed to being driven by changes to the stylus input.
The example we were really trying to build up to was this drip brush, where as you draw a line, drips automatically come off of the line
and fall down across the canvas.
But yeah, there's been other work we've done in that space.
Like you can generate really complex radial forms
where you're just doing one input
and then you automatically have
multiple radial branching forms
coming out from that one input.
It's a lot easier to show with videos.
In a prototype version,
we added the ability to actually do recursion where a brush could spawn show with videos. In a prototype version, we added the ability to actually do
recursion where a brush could spawn instances of itself. And then we had a termination condition
based on the number of brushes that have been spawned. We didn't actually use that in the final
version of the software because recursion can generate some tricky debugging scenarios. And so
we're still working on refining that part of the system but that's the overall idea yeah i i don't know why it just occurred to me
that i came up with a way to extend the level of abstraction here like in it's probably already
occurred to you you just take this but bring it into um like a vr 3d world where instead of making
brushes you're making like clay basically it's like a clay making
kit and so you can like make clay over here like you quote mix the clay programmatically with blocks
and then you like walk over or maybe one person's mixing clay the other person's like
shaping the clay so it's like it's like the same thing but for sculpting
yeah that's interesting i've talked with people about doing stuff in vr yeah more in a 3d
application we chose 2d originally because i love 2d art and also it's a lot of work to
to build these systems and any any corners you can cut in terms of simplifying the things you
can build is is a good starting point um but uh, we've talked about doing a 3D version of
it. I'm actually really interested in applying it, and we have done a little bit of experimenting in
this space already, to forms of... So one way of thinking about this is, I kind of framed it in
the lens of tool development. You're building your own tools.
But another way of thinking about it is reframing drawing within the space of interaction design.
So instead of actually making the artifact, what you're doing is defining the interactions you
want to have with the system that's going to then generate an artifact. That idea is something I
was thinking about or kind of came to me from these conversations I've been having
with the artist and engineer Madeline Gannon, who is probably best known for the gestural robotics
control she's done with these industrial scale robot arms. But she's actually doing all this
work from a standpoint of what are different ways in which we can think about digital fabrication interaction and control and what are the ways in which we might think
about CAD computer design not as being this process of sitting in front of a computer and
creating a design and then sending the machine to be fabricated but more of this process that
looks more like what happens when we make an object in a more traditional method like
she's really interested in on-body fabrication and clothing manufacturing,
where you actually might design an object
while you're working on a mannequin
or actually working with the person themselves.
And so she has this really powerful idea of,
can we think about interacting with some autonomous agent,
not as upfront we describe what it's going to do
with the artifact,
and then that's translated to tool paths,
and then the system executes it.
But instead of saying, like,
what are the class of interactions we might want to have,
or what are the ways we want this autonomous agent
to respond to us as we work,
and let's put the designer in the position of describing those,
and then being in the context of actually working with this machine
or autonomous system or agent to collaboratively,
or I don't know if collaboration is the right word.
Metaphors are important here.
I'm not sure I'm choosing the right ones,
but to finish the artifact.
So I've been thinking more about dynamic brushes
from the standpoint of what would it mean
if we're describing not how a digital brush is performing,
but how the head of a CNC machine or a 3D printer
is moving in response to decisions we make in the moment
of digital fabrication as opposed to ahead of time.
It maybe seems like a big jump,
but we've already done a version of dynamic brushes
where we've connected it to a CNC machine
and used it to create brushes
that actually draw in large format on the CNC machine and used it to create brushes that actually draw in large format
on the CNC machine.
The transcript for this episode is brought to you by Replit.
Normally, when I do a Replit sponsorship, I spend some time reflecting on the features
of Replit that I am most personally excited by and try to
write up something cohesive and organized so that I don't spend a bunch of time kind of bumbling and
making an ass of myself. But today I'm actually going to do that because I wanted to relate an
anecdote from earlier this decade, and it felt better to do that just off the cuff rather than writing something out. In 2016, my wife and I, having just recently been married,
decided to tour around Europe, living out of our backpacks, as is sort of the typical thing that
people from North America do when they get married. I spent a considerable amount of the trip
trying to learn Cloj learn closure because that was a
language that I'd recently become quite infatuated with and hadn't yet cracked my knuckles and done
a big project with it. So this felt like the right opportunity to do that since I'd be spending a lot
of time in hotels or riding on trains without much else to occupy me. The experience of setting up Clojure turned what would have been
a very exciting point in my life, where I get to dive into a new language, try a bunch of projects,
and make a bunch of cool things into several weeks of protracted misery. There were multiple
different build tools, and as somebody coming from outside the Java ecosystem,
I had no idea what palm meant or what maven was.
I had to spend a lot of time learning all of this terminology,
installing all of these different tools,
and making configuration file after configuration file,
trying to get it to work.
And after weeks of just trying to get Clojure set up,
I finally had something that let me actually build the project I was excited about, which, for the real heads in the audience, was
the first version of HEST. That early experience with Clojure colored my impression of the language
and the ecosystem in an irreparably harmed way. And I never ended up really embracing Clojure the
way that I originally thought that I was going to, and I've since basically stopped using it entirely.
So where does Replit come into this? On Replit, you can be up and writing Clojure in a matter of
seconds. There's nothing to install, There's nothing to configure. You just
tell Replit that you want to write Clojure, and then you write Clojure. If I had had that in 2016,
my relationship with Clojure may not have been adversarial right from the first time, and I might
still be writing Clojure, because it's a beautiful language. It is a wonderful work of
engineering, but for whatever reason, all of the design that has gone into it in terms of the
language primitives and the way that the technical decisions are grounded in solid conceptual design
work never manifested in the beginner experience. Replit did the work that the Clojure core team should have done.
So this is my way of sincerely thanking Replit
for creating something that solves a real problem
for newcomers to programming.
And that's why I mean it when I say thanks to Replit
for bringing us the future of coding.
I was thinking about how you said that you have like electronic circuits but
now they're also they like have a dual purpose and i was thinking about um the like part of what's
great about gestures is the embodied knowledge that we have but also just the joy of moving our
hands in ways that aren't just twitching our fingertips. It's just more joyful to move our whole arm.
But that's just our arm.
So I was imagining one day it's going to be like you're just going to dance.
And the way you move your whole body is going to be somehow mapped to some other creative form of expression that might be a physical object.
Yeah.
If you want it.
You could design it so that the more you bounce more you uh bounce like the i don't know the shakier you know you could i like
the idea of allowing people to design their own interactions based on how they have embodied the
thing they want to express yeah i i i'm really excited by that too i think this also gets to
the heart of like the the the challenges I'm most interested
in addressing right now and so one of the powerful things about programming is you can create these
arbitrary mappings between any input to any output you know within within reason but um and that was
actually one of the really nice things that came out of doing these studies with artists with the dynamic brushes system is actually had this moment of observing this painter have that realization where he was working with the system early on.
He was saying, I don't understand why you, you know, every drawing tool I've worked with you, I'm paraphrasing here, the stylus position dictates the brush position.
Like, why would you ever not want to map stylus position to brush position?
And then at one point he said, oh, oh, okay.
Actually, like I can transform the stylus position in some ways where I can use it to dynamically control the speed of the brush in addition to where it's actually located.
Or I can have it so that the position changes the
color of the brush that's interesting oh it's totally arbitrary how I map these things oh every
tool that I've used those relationships are arbitrary in in the space of software and oh I
can be a wizard I don't have to like take pigment and spread it around with matter.
I'm a wizard.
I can have it do what I want.
So that was really cool to see because, yeah, I think that's one thing I'm super excited about in helping people to create their own software and tools with programming is you can think differently about these relationships and these mappings. But it's also, the hard thing is then there are all these wonderful things about when you're getting to the space of mapping data from interactions that are
tacit or embodied or gestural, there are all these understandings that people have about
that, the way they move through space, the way they draw, that to do really powerful or specific or
precise things with them in a symbolic context, it can be very useful to understand them
from a different perspective, from a more discrete perspective, from a more formal perspective. So
artists think about, for example, the way they draw as a time-based act, many artists I've talked
with, but that doesn't mean that they think about it in the way that dynamic brushes represents time with regards
to drawing. That doesn't mean that they think about, you know, it takes me approximately 10
seconds to move this distance and I want to have this, you know, behavior in my drawing execute
on a certain time interval. There's a non-trivial process that goes between this
embodied or tacit understanding of drawing as a time-based process to how can I symbolically
manipulate that to do the thing that I want same goes for physics and I think I'm always sensitive
to the fact that on one hand there's a tension there and there's a chance for trade-offs or there's a
you know i i don't think you can preserve everything you get from one way of knowing
like a tacit way of knowing when you move into a symbolic or formal representation
but i do think there's power and utility there. So I, I, I, sorry, I just want to,
this is something that I was gonna bring up my myself in a few minutes. I'm glad we got here.
Now. What don't you think is possible preserving your? Yeah, which which one can't you go into the
other? Yeah, sorry, I just missed it. I think it's both ways, right?
You can't transmit from one to the other without losing something in the middle.
They're like...
Yeah, and this is maybe getting too abstract,
but any translation is an exercise in losing something, right?
There's value in it, but...
I guess, no, technically,
you could translate things losslessly.
I would say like from the perspective of,
I've been trying to write about this lately and it's been very challenging.
So I want to choose my words carefully here.
One way of thinking about it is, so, so earlier I talked about how drawing is this process of being a often speed and, and not thinking is an important element of drawing
where you basically artists train themselves to build up this physical intuition so that
they're not explicitly analyzing
everything that they're doing and instead reacting.
Like people who play sports.
And maybe, you know, I've also observed programmers
and people have talked about this
where they've reached such a familiarity
and level of skill with programming
that what they do is almost reactionary or intuitive in some senses when they're,
when they're developing some things. But I would say my experience with
programming is in general, it's this process of analysis,
of taking an idea and breaking it into smaller achievable problems.
And that is very different from this notion of reacting in the
moment. Almost system one, system two, the Kahneman's distinction. Yeah, maybe. I want to be
careful. There's been a lot of work in the psychology of
psychology of programming and also psychology and and psychological work and theoretical work around
tacit and embodied forms of working and I I would say my work could benefit by drawing
more heavily on that space although I am really inspired by things like the textility of
making or the reflective practitioner. Those works really drive how I think a lot about
some forms of expression that I'm interested in supporting.
I recognize the tension in my own work between
trying to combine something
that is often a fundamentally tacit
and embodied form of creation
with something that is often most powerful
as an analytical form of creation,
which I think programming is in many respects.
And that there are different pathways
you can take in that process.
The pathway I've taken is to try and select programming representations
that preserve some really important expressive aspects
of the passive form of creation like drawing.
But I also recognize that they're going to shift
some of the things that people are doing in that space.
So dynamic brushes, I never have claimed that it's something where people can just automatically learn the programming
language and start creating tools and drawing like they were before. No, like they spend time
learning the language. They spend a lot of time authoring tools, which they wouldn't be doing
necessarily with other systems. And there's more labor placed in that space of learning symbolic
programming and writing and
debugging the tools they're building and also understanding what are the trade-offs in that
space. Another approach might be to build a really expressive set of procedural drawing tools,
give people the ability to tune some parameters of those tools, but don't have them invest in the
actual process of writing the code itself. And there would be a lot of benefit to that approach.
And there have been things that people have done in that space.
But I think now I'm much more focused on this idea of, okay, recognizing these trade-offs,
recognizing that you have to think about drawing numerically.
You have to think about it analytically.
You have to think about it systematically.
What's the way in which we can help people make that transition more easily? Most of the work we're doing with dynamic brushes
right now is really around this idea of how can we show people representations of the drawing input
they're using in ways that help them to understand why a tool is functioning the way that it is,
why a state change occurs when it does, and help them make
informed decisions about the process of programming. Yeah, it's a trade-off.
Yeah, I definitely see the trade-off of, I do a lot of thinking in this area too,
but I'm going to be a little bit less careful with my words. I'm just going to like kind of
talk around it to evoke what I'm getting at i guess i see on the one hand
there's things that humans already know like we have a very embodied sense of gravity we're not
born with it but i think over the first five years like a pig kind of way like we get it you know
like oh okay like i drop things i expect in the fall it's like you feel in your body and you feel
it and you expect it in objects you manipulate but then it takes a long time a lot of formal training to get someone to transfer that
to a knowledge of physics like an abstract knowledge of physics there's like it's it's
quite a quite a lot of education that has to go to teach that formally and i guess maybe a better
example one that's kind of close to my heart, is I was a logo kid.
I was a math phobe who was exposed to logo.
And I learned how to draw a circle.
I kind of eventually intuited on my own, oh, wait, if I just walk up a little bit and turn a little bit and do that 360 times, I got a circle.
Then when I eventually went to calculus, it was just so easy to understand what a derivative was because I've walked along curves before.
Anyways, what I'm getting at is, one way to put it, is Alan Kay has this critique of Scratch that you may have heard of.
Yeah, yeah.
I guess you haven't worked on Scratch, but I feel like you're maybe one of the better people I've had on the podcast who could defend it from Alan's criticism.
Basically, there are certain human universals
or things that humans are good at like drawing and singing
and dancing and playing
but then there are certain non-human universals
that are not more enlightened
but just they're also very desirable
but they don't come naturally
they require a certain kind of investment
but then there are these interesting ways to get humans
from the universals, the things they already know
and love and like transfer leverage that love and knowledge into a new, more powerful, but also harder domain.
So anyways, I'll let you take it from there.
Yeah, I mean, that makes me think of, and maybe this is what you're drawing from, like Alan Kay's essay on, I think it's a user interface,
a personal view. I'm not sure if I'm getting that right, where he talks about this notion of
how you can move between more concrete forms of understanding to more symbolic forms of
understanding and symbolic knowledge versus visual knowledge. This is coming, I think,
from Jerome Bruner's theories about different forms of knowledge, and I'm not representing
it in its entirety but this
idea that you might be able to yes that like symbolic knowledge being this really powerful
form of representation but maybe less natural for people to understand but visual and kinesthetic
knowledge being more familiar and more natural but that by working in the visual and kinesthetic
domain you can help scaffold the process of understanding and manipulating symbolic knowledge i'm really excited by that idea and i think it's something that i would say i need to do a better
job with some of the systems that i do in thinking about so much focus initially has been on how do
we select an appropriate representation for a given domain with still the knowledge of there's there's
going to be learning challenges or conceptual challenges here and what are the ways in which
we support people in making connections or or moving to really understanding the symbolic domain
when they're starting with a visual or kinesthetic form of manipulation. And I think there's a lot that I can do,
not hiding information, showing information visually.
There's also a lot of huge challenges.
So one of the very concrete challenges in dynamic brushes
is we have people working in a geometric domain, 2D drawing.
And then we have this type of data that is
geometric. It has natural geometric properties, the stylus input in many ways. And we have these
other data types that you can map them to geometric properties, and it's very powerful,
but they're more abstract. They don't inherently have a geometric quality that aligns with the
geometric properties of the 2D canvas. So like a sine waveform.
I can draw a sine waveform on the canvas,
but that's also misleading in some respects because the dimensions of it are going to be arbitrary
relative to my drawing.
And really that data is,
it's also very powerful to understand that data
more abstractly or numerically.
And so there's this idea of, okay, yes,
I want to help people draw those connections.
I also don't want to mislead them. I don't want them to have an incorrect intuition about how to think
about this type of input and this data. It would be a problem if the painter got the idea that,
oh, there are certain types of relationships I'm supposed to create with data in dynamic
brushes and other types that I'm not supposed to create. So I don't think I have answers to that,
but it's a topic I'm really interested in.
One of the big challenges I see in that
is the more programmatic of an interface you expose,
the less constancy or repetitiveness they can expect.
So the harder it is for them to build up
those somatic intuitions that are so important. Part of what makes, allows artists to get really good with a paintbrush
is that paintbrushes are pretty similar, but you're allowing them to create all these new
paintbrushes. So I would expect, or one solution to that problem would be they spend a lot of time
iterating on what paint brushes or what uh you
know uh dynamic brushes they want to make and then they kind of maybe will leave it kind of like it
is and then make a bunch of art and then the next series they'd like modify the brushes again because
if you modify the brushes too much it's i could imagine that it's hard to build up a certain
intuition of like what different gestures actually
accomplish yeah and that's the behavior that we've observed so far with people using the system is
there we'll spend a lot of time in the programming environment kind of experiment drawing and then
a lot of time in the drawing environment and really only making minor tweaks if anything to
the brushes but i'm not saying that's wrong I also think that we're working
hard to find ways to enable people to move more easily back and forth between the two spaces
because there are ideas that emerge during the process of drawing that you want to be able to
implement in in programming and to be able to translate to the programming environment
it's not always easy so we've done really simple, obvious things like you can
record some drawing input and loop it back through the system while you modify the code.
So you don't have to like experiment every time you can immediately see how your changes
update the drawing. But there are, there are bigger questions that I know have been explored
by other people in other domains, but this idea of, Oh, do I ever want to be able to,
to do something in the drawing environment that actually describes or manipulates a relationship in the program?
And Brett has done some really cool stuff in this space. of how do I distinguish between what I could do that's possible versus what makes sense
and what has like a meaningful semantics.
You know, like I love the idea
of having people be able to draw a waveform
in the drawing environment.
And then, you know, that's a great idea.
You know, less clear, like I've talked about this a lot
with people on the the scratch team around like
how would you geometrically describe a conditional or not you know you could come up with an idea and
there might be some cases where it makes sense but there might be a lot of other cases where
it might not make sense so yeah a lot of challenges in this space one of the things that
makes this very concrete for me this idea of like a standard interface versus a programmable interface
is um 10 years ago my dad got this like whole new tv system and it came with this
touchscreen interface and he could never turn on let alone use this tv and the newer interface
like the state of the art of today is made by the same company but it's a um it's not a touchscreen
anymore it's like back to an old-fashioned interface where you actually specify to the
company what you want printed on the buttons and they print it onto the buttons uh so it's like
the same amount of programmatic programmability but um but once you've set it up they print it onto the buttons. So it's like the same amount of programmability,
but once you've set it up, they print it in a way that makes it much more like my dad can
not only can turn on his TV now, but he doesn't have to look at the remote to know where the
buttons are. That's interesting. Yeah. Yeah. It's, I think there's need for more work in the domain of creativity support and systems research.
I think looking at some more longitudinal impacts of use patterns and learnability for some of the tools we build over time. Like, I think there's,
in general, this idea that people should learn a system really quickly or use it really quickly,
and if they can, then it's a good system, and if they can't, then it's a not effective system. But
that's not necessarily the right metric for how we think about how we might support people,
you know people using software
or using a programming language or a creative tool.
And some of my favorite things come out of looking at
what artists who have been using Photoshop and Illustrator
for a really long time do with it,
where they use the filters in ways that I had no idea
that you could use them in those ways.
And they'll call them hacks.
They'll be like, oh, I use this hack.
It's not really how it's supposed to be used, but I get this effect by
combining the blur filter with the, I don't know, I'm not going to remember the filters. I never
use them. And it's interesting because it's like, oh, you could see that as an unintended use,
or you could see that as this is what happens when people use software of an extended period of time, they adapt it to their own applications. And that's a really
powerful and valuable thing, but it's really reflected in how we present and evaluate systems
from a research perspective. And that it's a system we build over a year to six month period,
and we test it for hopefully a while. And then we kind of understand what people can do with it in that timeframe.
But not what people might do with it
using it over a year or two years.
There's only exceptions to this, but.
I have a very funny,
I guess, illustration of this kind of dialectic.
I've been having this silly conversation
with a stranger on Twitter.
His position is that programming
should be as intuitive as possible.
And my position is that it's not necessarily a bad thing
if it requires certain investment,
but that investment pays off in power down the road.
Yeah.
And so he sent me yesterday a video of a monkey,
like an actual ape, using a touchscreen.
And he's like, see, programming should be this intuitive
that even a monkey can use it.
And so I sent back a video, like a timeless video video of someone who didn't know how to play piano.
At the end of the year, he could play really intense piano.
It's like, no, no, this is what programming should be like.
And I think it's a funny dichotomy of the two different views of immediately usable versus rewards investment.
Yeah, I guess if you think of programming from the interface or
environment perspective, yes, we should make things easy with an important asterisk there,
which is like easy without restricting the important forms of power expression that we
want to deliver. But if you think of programming as a way of thinking, it is almost like not
relevant to think about like, how do we make
this way of thinking easy? Because it's more about, you know, we want to provide pathways
into engaging this way of thinking, but you can't trivialize the difficulty of learning
abstraction. You know, I remember when I first really understood the concept of object-oriented
programming, it's a powerful idea. And I don't
think you want to say, well, there are things that are hard about object-oriented programming.
Let's choose a simpler idea that's more intuitive, more obvious to people. And you're diminishing
the power of that idea, you know, and arguably like object-oriented programming is more aligned
with how people might think about things in other programming
models, but there's value in understanding and learning different ways of seeing the world,
and programming offers that. I don't want to diminish that. And also, the idea of what is
intuitive, I use it a lot, so guilty of this, but we don't really have a good grasp of, I think,
in many ways, things that are intuitive, Because what's intuitive for one person could be totally different from what's intuitive to another person.
And the notion of dynamic brushes, we chose this representation for drawing not because we thought, oh, this will be the most intuitive representation for drawing.
Because this is how people think about drawing.
Because every artist I talked to thought about drawing in a different way.
There were some similarities that we tried to recognize,
but overall the idea was,
no, if I build a programming representation
for how I think about drawing,
it's not going to correspond
with how this other person thinks about it.
It's an impossible design challenge.
So I think there are other principles we can involve
in terms of what does this let people express, what does it limit them from expressing, and what are the reasonable scaffolds we can provide to helping them learn it.
We want to think about things in terms of correspondence and how close the mental model of a given audience is to a given representation, but that's not always feasible. And I would argue in programming,
it's often not feasible.
Oh yeah. I'm glad. I'm glad we, I don't know. I like,
I like that answer a lot. Um, uh, and that,
I didn't know if that's what you were going to say. Um, I,
I thought maybe you'd come out on the other side of like, no, no, no,
it has to be as easy and natural as possible.
Yeah. Easy as possible, but no easier.
And I'm ripping off multiple people with that statement.
So back to some of your work, I liked your discussion of the evaluation of it.
You talked about how tricky it was. And I think it's kind of, it's similar to whenever you make a tool, it's hard
to evaluate a tool because you have to evaluate how people use a tool and what artifacts they
produce. So it's like, we are one step removed in a way. But I think for a creative tool,
it's even harder. Like, so I'm someone who designs tools for people to build user interfaces.
And so I, there's this project called seven GUIs where there are seven user interfaces and I just can watch people build those seven interfaces in
my tool and call it and call it a day. But, um, when you're building a tool for artistic, uh,
creative expression, you're, you're building a tool for people to like purposefully break
boundaries and do things that have never been done before. So's like so it seems really hard to evaluate that sort of a tool yeah it is hard uh so one of the the best metrics that i think mitch and leah both held up and i've
similarly tried to pursue this metric in my work is like you know a system is successful when you
see multiple people making things that
you didn't know were possible in that system it's great if if if people are able to make complex
things or able to easily do the things that you thought the system would be good for but if someone
uses it to then do something you had no idea was possible then you've succeeded in some way from a creative standpoint.
How you design for that is not obvious. I think you just designed with a blindfold on.
I couldn't have imagined they did that.
I wasn't looking.
Yeah.
Jay Silver has this notion of developing a tool around a project space. So if you develop a tool that's
able to create one type of project, then you've got a point. If you know that the tool can create
these two types of projects, then you've got a line and you've got a spectrum of projects across
that line. If you know a tool is able to create these three types of projects, then you've got
a space and you've got a larger area of possible projects that might emerge within these outer parameters. And he describes this in
his dissertation. And I think that's sort of one way of approaching both the design and then
evaluation, where you set up, and a lot of the work that I've done is sort of gravitated toward
this, you try to choose these targets of the types of expression you think your tool should be good for,
but pick them in a way that they are different enough
that you're hopeful that there will be
some interesting variation in between them.
You design for those,
you inform that through different processes,
looking at what people do while you're building it,
and so on.
And then when you evaluate it,
you also think about,
who am I selecting that corresponds with or diverges
from these forms of expression
that I've tried to design for
and then looking at what they do
and how it aligns with these points
you've kind of created versus unknown areas
or points between them
and really trying to look at both
what you've set up as sort of the types of
things you think the system enables making and also what people make with it and how close or
how far it is from those things. So that's one reason that I've also tried to have people use
the tools that I've built over a longer period of time is because you get more interesting artifacts
out of that and you have more data with regards to the forms of expression that are
possible by analyzing those artifacts. It's definitely a qualitative method of evaluation,
and there's a lot of noise in the data, so there are limitations to it. I recognize that.
One thing that I'm trying to move towards is actually scaling up some of these studies,
and so we're working right
now on releasing dynamic brushes as a stable app for people to use, although that's a long process
so that we can look at more generally, what do people do with the system independent of a formal
study or in the wild, quote unquote. And that's really inspired by some work that Leah and
Benjamin Mako Hill, who's professor at UW, did with the LilyPad,
where they looked at what people were doing with it
over the long term in actual practice
by looking at sales data and work posted
to different online communities
and doing more of a longitudinal and data-driven approach
to analyzing what are the creative affordances
of this specific platform.
But I'm always interested in other forms of evaluation,
but I think that fundamentally,
if you're designing for creative expression,
you need to accept that there are trade-offs
in that traditional metrics for evaluation,
efficiency, speed.
Those are great, but they won't work.
They won't tell you the whole story here.
Yeah, and that
was one of my questions can can people use these tools exciting to hear that they'll be coming soon
yeah so para might be coming in a different version than than our original prototype
dynamic brushes we've been building it we're working on another version that has these
debugging support features in it
my goal is to get in the app store with the the release of that work um it's another challenge of
of academic research is it's often not incentivized to release your software as important as that is
so i'm trying to find my own strategies for for doing doing that. Um, often the paper is the artifact. So.
Yeah. Yeah. Well, I guess this is kind of, uh, dovetails with another question I have the, the, the, um, uh, interplayedness, interplayed, interdisciplinary, interdisciplinary.
That's the one. Nope. I'm just going to skip the words.
The systems engineering method that you did where you do this need finding research
and then you build a tool and you watch people use it you evaluate it then you kind of repeat
and build a new tool it feels a little bit rare to me like i didn't think that academia would
incentivize this kind of work it seems like academia it doesn't seem to support need finding
as much i'm kind of I was excited to see that
that was like a part of your process. Yeah, it does and it doesn't. I mean, I've been fortunate
to work primarily in the field of human computer interaction where it's a big enough community
with a diversity of thought. So there are definitely people who are more interested in really quantitative
evaluation, forms of measurement that are less noisy. But there are also huge groups of people
who really value tackling messy problems and also approaching it from a human-centered design
perspective, which need finding is a big component. These
things take time. Building software takes time. Testing it with people over a longer period of
time takes time and resources. Yes, that's not always supported in an academic context, but
I'm also encouraged by the fact that there are increasingly in computer science research emphasis on things like open source or verified
artifacts or functioning tools as a contribution and as something that we officially recognize.
And so my hope is that there will be more official incentives for doing this work in addition to the
knowledge gain that we can get from it.
And yeah, I'm also excited to see as I continue in my academic career running a lab as opposed to working as a student, what opportunities I'll have for more collaborative forms of production
and trying to scale up projects over a longer period of time and with more people involved in
them. What kind of like Scratch, I guess, is like the big example?
Scratch, I think, could only happen at the Media Lab.
Scratch is a very unique example in that it's basically a small company being run inside a research group.
And I think it's unlikely that I would...
Could pull that off, you know.
Yeah, like Mitch has made really
good choices in that respect but he has you know made a decision to um like he wants to get
programming to as many children as possible which I think is a really laudable goal and and scratch
is a an approach to doing that I think I'm maybe more leaning towards
one of the benefits of academia
in that you can explore many different directions.
And so I want to release software
and I want to think about building systems
that have a lifespan beyond the paper.
But there are many ways to do that.
For example, I've had great partnerships
with Adobe
and industry, and I'm really interested in pursuing those going forward. There's trade-offs
due to that, but so far that's been a good option for me. And there's also opportunities to
contribute to other creative coding communities that are more focused on specifically putting
systems out in the world. And so the work that the P5.js and the processing
community and also the open frameworks community does is incredibly inspiring and valuable. And I
also think about ways in which in academia, I might be able to, in computer science academia,
make more connections between the work that's being done in the creative coding community and
the work that's being done at conferences like CHI and human computer interaction research and programming languages research
and how there can be an exchange of ideas but also an exchange of resources in some respects.
Cool. Well, I think we have taken enough of your time. No, this is great.
I really enjoy hashing through these ideas.
Clearly, I have a lot to continue to hash through,
but it's been really fun to talk with you about this.
Yeah, yeah, it's been great having you.
Yeah, definitely.
Love being connected with new people,
thinking about similar ideas. And it's always encouraging to talk with people
who are interested and
value the notion that programming can be
better by building different programming
languages or different
types of programming. We can open it up
to new groups and new people.
That's
not always an idea that everyone believes in.
It's good to be reminded
of it.
Before we sign off, I wanted to give you an opportunity to just list some of those places on the Internet where you could be reached in different ways that you wanted to be reached if you were looking for collaborators or anything like that.
Yeah. So I guess very concretely, if any of these ideas are exciting to people, I'm starting up a research group at the University of California, Santa Barbara in July as an assistant professor. And so
we are actively looking for people who are interested in potentially pursuing these ideas
under a PhD or a master's degree, but also visiting researchers, visiting artists. So
if you're excited about this and you're interested in talking more, definitely reach out to me over email, jmjacobs at ucsb.edu. But you can also find me on my website, which is just
jenniferjacobs.co. Yeah, I think that's probably the best way to talk with me, but I'm also on
Twitter at JSquare. That's it for our show today. to steve for having the foresight to interview jennifer jacobs
i was actually gearing up to reach out to her and ask her to come on the show myself when i learned
that you had already done this so that was very fortuitous and i um i was delighted to hear this
interview there's a not insignificant chance
that I will probably at some point
ask Jennifer to come back on the show again
just so that I can ask her a bunch of questions
that occurred to me while listening to this show.
That's the real benefit of having a podcast like this
is that I get to reach out
to all of these interesting thinkers in our field
and ask them whatever I want to ask
them. And so on that note, if there are people out there that you would like to hear me interview,
I have a fairly long list of guests that I've selected already. And so not all of your
recommendations will be people that I can eventually bring on, but there are many people
who would be a perfect fit for the show
that I've never even heard of.
And so if there's anybody out there that you know of,
and I'm just going to explicitly say
extra, extra points
if they are from an underrepresented group,
since I am very committed to having this show
reflect those
principles of inclusion and representation that I spoke to earlier, please tell me who they are
so that I can bring them on and share with you all of their wonderful thoughts and visions for
the future and wild ideas. The next episode is going to be an interview with Ravi Chung,
one of the key figures in the Sketch and Sketch project, which Jennifer mentioned in this interview and which has made waves through our community in the past.
Ravi and I had a very long interview. It was something like three hours and 45 minutes.
So I'm either going to cut it down significantly or split it into a two-part.
So this one might take me a little while yet to finish editing and get out.
So if there's a bit of a delay between what you're hearing now and when the next episode is released, that's why.
You can find the show notes at futureofcoding.org slash episodes slash 48.
You can join our Slack community at futureofcoding.org slash community.
I'm Ivan Rees, and I will see you in the future. Продолжение следует...