Science Friday - AI Conversation, Robot Trust, AI Music. May 18, 2018, Part 2

Episode Date: May 25, 2018

Should autonomy be the holy grail of artificial intelligence? Computer scientist Justine Cassell has been working for decades on interdependence instead—AI that can hold conversations with us, teach us, and otherwise develop good rapport with us. She joined Ira live on stage at the Carnegie Library of Homestead Music Hall in Pittsburgh to introduce us to SARA, a virtual assistant that helped world leaders navigate the World Economic Forum last year. Cassell discusses the value of studying relationships in building a new generation of more trustworthy AI. Robot assistants talk to us from our phones. Home robots have faces and facial expressions. But many of the robots that might enter our lives will have no such analogs to help us trust and understand them. What’s a roboticist to do? Madeline Gannon, a Carnegie Mellon research fellow, artist, and roboticist for NVIDIA, trains industrial robots to use body language to communicate, while Henny Admoni, psychologist and assistant professor of robotics at Carnegie Mellon University, teaches assistive technology to anticipate the needs of its users.  The pop hits of the future might be written not by human musicians, but by machine-learning algorithms that have learned the rules of catchy music, and apply them to create never-before-heard melodies. Those tunes may not even require human hands to be heard, because a growing army of musical robots, from bagpipes to xylophones, can already play themselves—even improvise too. We talk with computer scientist Roger Dannenberg and artist-roboticist Eric Singer about the implications of computerized composition, and unveil a song created by AI. (We’ll let you judge whether it’s worthy of the top 40.) Subscribe to this podcast. Plus, to stay updated on all things science, sign up for Science Friday's newsletters.

This is Science Friday. I'm Ira Flato, coming to you from the Carnegie Library Music Hall in Pittsburgh. Yeah. You know, bit by bit, artificial intelligence is working its way into our lives, and it's not just self-driving cars. Look at the AI assistants that live in our smartphones like Siri, or the ones that we've added to our homes like Alexa. Someday, maybe soon, AI may be making phone calls to schedule your doctor's appointments, or even helping your children in the classroom with social skills or creativity.

But for AI to be most helpful to us, we have to trust it, trust it enough to share our personal secrets. And that's where my first guest comes in. She spent her career studying how we build trust with each other and then with virtual personalities. And the key, she says, isn't necessarily building knowledge, but rather asking questions, making small talk, and opening up to one another.

Justine Kassell is Associate Dean of the School of Computer Science and former director of the Human Computer Interaction Institute at Carnegie Mellon University. Welcome, Justine. Thank you. It's great to have you. Now, I know your work focuses on artificial intelligence that has a visual component like an animated face.

Why not go straight to making physical robots with bodies on them? Well, it's true. My work is more like cartoons on... a screen, and that's because we can still do more realistic kinds of movements with a cartoon than we can with a robot. We have a video that actually demonstrates one of your projects called Sarah, the socially aware robotic assistants. Before we look at it, tell us a bit about.

So Sarah, a socially aware robotic assistant, is a personal assistant. We were asked to create this for the World Economic Forum meeting in Davos in two. 2017, and so more than 300 world leaders interacted with Sarah, and Sarah helped them meet other people at the same level of importance as they are, and help them find sessions to attend that matched their interests. Doing that required them to disclose information about themselves, and in fact it turns out that self-disclosure is one of the ways we build rapport with other people. So she had to get them to self-disclose, and she did that by asking questions about them,

but also by self-disclosing about herself. Like, gee, this is my first time here. It's pretty awe-inspiring. I'm kind of freaked out. Are you freaked out at all? All right, let's go take a look. Let's go watch the video and take a look at how that happened. Hi, I'm Sarah.

It's such a pleasure to meet you. Might I ask your name? Hey Sarah, my name is Fetty. Nice of you to take the time to stop by. So how are things? It's going pretty good. It's going pretty good. Great. Well, let me introduce myself.

I've been asked to play matchmaker by helping attendees find sessions to attend and people to meet, based on their interests. All you need to do is tell me a little bit about yourself and what you want to do while you're here. you're here. So I'm a master student at Carnegie Mellon University in the computer science department and my research focuses on artificial intelligence and natural language processing. And I'm mainly here just to network, meet people and kind of the tech, the tech scene and maybe go to some interesting sessions. I've never met someone like you before. It's refreshing. Let's see if we can find some interesting people to meet. Just one minute while I look this up. There are details on the screen about Justine Castle. I feel you share some interests. Yeah, Justine

to sell. I've heard of her. It would be nice to meet her, though. I can either send a message on your behalf suggesting a meeting, or I can give you the information and you can set up the meeting yourself. Would you like me to send the message? Yeah, please do. Oh, good. You'll see a message from me in your Toplink, private message inbox introducing you to. Awesome. Awesome. Awesome. You know, it was, yeah, that's a great, great video. You know, it would seem to me from looking at the video is she wasn't just talking. She was doing a lot of things at the same time.

Right. There was not a real script that she was reading from. No, she's actually adapting her behavior to the behavior of the person she's interacting with. And what we saw was that she's automatically sensing what conversational strategy the people are using. Are they self-disclosing? Are they praising her? Are they teasing her, violating social norms in some way? And then she adapts what she says in order to raise rapport. And you could see rapport rising.

And what's interesting about Sarah is that it's the first project that considers both what's going on in the human and what's going on in the computational agent in order to figure out whether rapport is in place. Well, we talked about having to gain trust from a computer. That seems to be a key. You keep mentioning over and over again, rapport. Yeah. Right?

You have to build a rapport. So that's where the trust comes from. Trust does come from rapport a lot. Yep. And so why do we need a physical face on Sarah at all up there? Does that help build that trust or rapport? It's a really good question. I'm really not for embodied agents just because they're cute or cool,

but because we use our bodies in ways that demonstrate how we feel about the other person, where we are in the conversation. And so we need bodies on agents like this because in our conversation, and I'm certainly an example. I'm trying not to knock this microphone off while I just, We use our hands and our face and our torso to indicate who we are to the other person, who the other person is to us. So that body language.

Body language is important. So how is Sarah a step up from, say, Alexa, you know, or Siri or any of those other? Where shall I begin? Let me count the ways. So Sarah, first of all, does have a body, unlike alexer. unlike Alexa or Cortana or any one of the current personal assistants that are out there made by these major corporations. And she uses her bodies in the ways that I just described to build trust, to build rapport, to build confidence in what she says.

But she also, unlike those agents, is while she's talking, she's automatically sensing your response. So she's looking to see whether you're smiling, you're looking at her, your eye gazes towards her or away from her. And she's using those cues as well as what you say to decide how you feel about her. What's hard about making AI that can interact realistically with humans? What's the hardest part? We are unpredictable as humans. You think? Yeah.

Long years of study before I used. understood that. And Sarah has to be able to keep up. She has to be able, in real time, to do what we call classify, that's a machine learning term, classify the conversational strategies you're using. And so you saw her doing that. She chooses one of eight strategies. That's really hard. And you need a lot of data in order to get it right. But the kinds of phenomena that I'm interested in, these things like being polite or violating social norms don't happen all that frequently in conversation. So what you've all heard of referred to as big data,

these kinds of phenomena, the ones that are really important in interaction, are small data, and that makes it hard. It's interesting. You show us what was going on at the World Economic Forum, a bunch of a lot of rich people up there in Switzerland. But let's bring it down to Earth more. what kind of practical solutions can you use Sarah for? We can relate to it.

Yeah. So Sarah is a platform, in fact, a set of computational modules that work together that can be used for really any purpose, you know, as long as you put them together in the right way and add a kind of a functionality. And so we have a system called RAPT, the Rapor-Aligning peer tutor, who looks like a teenager. and who peer tutors teenagers in linear algebra. And it builds rapport in order to help the student learn.

And we know in human-human interaction, and all of my work is based on looking at real humans in interaction with other humans, we know that when kids feel rapport with another kid, they learn more. And a second thing that we're building is called Skipper. big into acronyms, as you can tell. And Skipper stands for sensing, curiosity in play, and responding. Took us a long time to come up with that one. And Skipper is for younger children, eight or nine-year-olds.

Now, in the contemporary school system, there's a lot of teaching to the test because today in America, schools get funding as a function of how well the students do. And teaching to the test is pretty much antithetical to curiosity, to passionate curiosity, like you and Einstein. How do we keep... Don't put us in this same. I'm trying to get on your good side. Bill and I would be close enough, you know. So how do we get kids to remain curious about their world?

Because really curiosity is the key to exploration, which is the key to exploration, which is the key to learning. And so Skipper doesn't build rapport. It builds a sense of curiosity in groups of children by watching what they do and capitalizing on those of their behaviors that are leading them towards being curious. It's the first project where we've worked with groups of kids, so it's not just a two-sum, it's three to four kids. And first time we've worked with curiosity. How do you know it's Successful. How do you know what's working? That's the million-dollar question. We have to evaluate all of this. Now, we're in the process of testing wrapped, the teen linear algebra tutor,

with, so far I think we've worked with 50 teenagers, and what we're looking at is their learning gains. That's the only thing that really matters. Who learns more? The kids who interacted with the version that builds rapport, or the kids who interacted with the other version that doesn't build a point. I wish you great luck. Great success and good luck with your work. Thank you so much. Justine Kassell, Dr. Kassell,

Associate Dean in the School of Computer Science and former director of the Human Computer Interaction Institute at Carnegie Mellon University here in Pittsburgh. Thank you for taking down to be with us. After the break, we'll look at building robots that we can trust and empathize with and where art and psychology fit. I'm Ira Flato, and this is Science Friday

from W1YC Studios. This is Science Friday. I'm Ira Flato coming to you from the Carnegie Library Music Hall in Pittsburgh. My next guest aren't working on anything that looks like you and me. This isn't Dolores from Westworld or even the Maria from Metropolis, but perhaps you share an understandable concern that robots, as I say, are going to replace us. That's one of the themes we see a lot in science fiction, for example, or all those movies about robot uprisings we've been watching over the years. And even with completely benevolent, self-driving cars,

safety and trust are huge part of that conversation as we've been talking about it. So how do we build trust? Now, how do we make the case for robots as partners for people doing the things they do best and leaving us free to do what we do best? That's some of the heavy questions my next guests are talking about. Madeline Gannon is an artist, a roboticist at Nvidia, the chip company,

and a research fellow at Carnegie Mellon University, she's sometimes called the Robot Whisperer. Welcome. Thank you so much. Thanks, fans. I'm happy to be here. And Henny Edmoni is a psychologist, an assistant professor in the Robotics Institute

at Carnegie Mellon and Director of the Human and Robot Partners Lab. Welcome to you also. Thank you very much. Thank you very much. Thank you very much. And, Annie, let me start with you. Tell us about the robots that you are working with. Great.

So the work that I do looks at building assistive robots. So when I say assistive robots, I mean robots that help people. And typically when we talk about assistance, we mean helping people who have some sort of disability or some sort of inability. So one example is a robot arm that attaches to a powered wheelchair. This is a commercial product. It's made by a company in Canada. And this robot is designed to help people who have upper body motor impairment

to do everyday tasks like picking up a glass of water and taking a drink. Now, this robot is really challenging to use because it has to be teleoperated. I look at how do we add intelligence into the system, use artificial intelligence, to make these robots more assistive for us so they can predict the help we need and actually assist us toward that help. And what's really fascinating because I've watched your work, I've watched what you've been doing, is that the way you are getting the robots to respond is you control their arms by using eye gaze. In other words, the robots are watching where you're doing.

eyes are going. Right. They can pick up on that? Right. And what's very important to note is that they're not eye remote controls. So there exists in the world of assistive robots controllers that just use your eyes, right, where you look at an object for five seconds and then the robot goes there. And those are very useful if you have no motor control.

But that's not actually what we're trying to do. What we want to do is capture the natural human behavior, the eye gaze that people are producing all the time, in order to predict better what they're trying to do. produce all kinds of nonverbal behaviors when they're interacting with their world. One of the nonverbal behaviors they produce that's very, very informative is eye gaze, where they're looking. And we actually have a video. Let's play the video.

You can watch the robot in action. Right. It's amazing. Yes. So here we have a video from a study we did recently where we asked people to control this robot arm to pick up marshmallows from a plate using the fork in the robot's hand. And so our participants control the robot through a joystick. And we were also monitoring their eye gaze through an eye tracker.

You can see the participant is wearing a headset that kind of looks like a pair of eyeglasses that has a camera facing in at her eye and a camera facing out at the world and it's looking at what she is looking at in the world so we can know where she's looking. This is not an easy task. This is actually a case...

In this case, the robot's actually helping her do the task and it still takes quite a long time to pick up a single bite of food. You can see videos and photos of my guest's robotics work on our website at ScienceFriday.com slash robotics. Does the robot anticipate what she's trying to do? It's anticipating what she's trying to do through her joystick control.

It's not yet using her eye gaze. This was for us to understand how eye gaze, what the eye gaze dynamics look like in this kind of task. So does the robot have to understand human psychology, thinking along the way? In this, it will eventually. In this particular case, it doesn't.

It will be more powerful when it understands human psychology. And what I mean specifically about that is understands how people use their gaze when they're interacting. So psychologists, for example, know that when people go to reach for an object, if I were going to go reach for this water bottle on the table, I would move my eyes to it and look at it about half a second before my hands even started moving. And we can pick up on that with a system like this to actually make it predictive.

That's incredible. Now, Madeline, you're an artist who reconfigures robots. How does that happen? Yeah, so the robots that I tend to work with are these big, giant, monstrous industrial robots. So these are the machines, the one-ton machines that make your cars, for example. And what I try to do is to get those machines to do things that they're not intended to do, not designed to do.

They're designed to do short, repetitive tasks over and over 24-7 for their whole lives. That's a very boring life for a robot. So I try to find ways to sort of liberate them from their constrained environment and find ways for them to interface with people. and for people interface with them. My dream is to take that machine that has superhuman speed and superhuman strength and to give me those superpowers. That's the sort of software interfaces that I want to have.

Finding ways for this machine to enhance, augment, and expand what we do, not replace it. For a little bit more context, this was an invitation to bring an industrial robot to live at the Design Museum in London for six months. So Mimas got to have a holiday off of her, assembly line and got to live inside a museum and play with about a quarter million people over the course of six months at the design museum. It looks like the kids are treating like a zoo

animal. You know, finally they're trying to get its attention, it's behind glass like it would be at the zoo. So how does it understand what, you know, the kids are doing, what all these people coming up to do want it to do, or how does it learn how to react? So that's a little bit of the design part of it. The way that I approach this is Mimas is a, is a, is a, is a, a, you know, big inanimate object. It's a robot. It should move robotically. Instead of having it move like a robot, I try to make it move like an animal to see and respond to us in a lifelike way. The intelligence behind this machine is nothing that doesn't already exist. So it's an off-the-shelf piece of hardware that already exists in factories. The sensors that I used are $35 Microsoft

Connects that you can buy off of eBay. And the only thing that is bringing this machine to life is a little bit of clever software to duct tape everything together. And for me, it's really important to show that the way that our automation systems are today are not set in stone, that these are choices, sometimes inadvertent, sometimes deliberate choices, that our systems of automation exist, but that we can have a future that is a bit more preferential and inclusive to people. Yeah, Henny, you were nodding here. You like that. I love that, yeah. What did you like so much about it?

I think it's fascinating that this big, bulky, inanimate object can have so much personality just by the way it moves. You know, and we add a lot to that. I mean, I'm so happy to be sitting next to R2D2 on stage. That is one of my favorite robots. He has a companion robot in Star Wars. I don't know if you guys like science fiction. I'm assuming you do.

But C3Pio is a humanoid robot, and we were just chatting before. It's like, you know, C3Pio, you just leave them in the desert. okay. R2D2, you go back for him. You make sure he makes it, but the mission gets complete with R2D2. But you're both talking about giving the robots a personality. You're definitely talking about the personality. And you're talking about when you create your robots, you want them to have a personality also, to a certain extent. To an extent, yeah, and I think it's fascinating that they do anyway. We can't help as humans but attribute personality to our inanimate objects. We do it all the time. We do it almost unconsciously. And I think that robotics and human-robot

interaction really benefits when it drives that attribution of personality and attribution of animacy in a systematic way. So another way to think about personality is legibility, that we're building some transparency for what this robot is intending to do because it's broadcasting it to us. In one way, it makes it a more palatable experience with the robot. But on a very pragmatic level, it's just being more transparent and legible and communicating what it's about to do to us, the same way that other things that are in our environment like animals, for example. We can read their body language and know what they're about to do. We're talking about personalities. One thing, one thing I noticed immediately that was missing from your animals. What does zoo animals have that this robot

doesn't have? A face. You could have put Bozo the clown on the end of that thing or some sort of mask or something, but you chose not to do that. Why is that? That was very deliberate. So Mimas, I did experiment with some things, putting mirrors or cameras on the robot. At the end, I just decided to leave her naked as she was birthed in this world. And it's incredible because, you know, for robotic arms, they're only as useful as the thing that's on the end of them. Their end effector, so a gripper or welding tool. And for many people who are in the world of industrial robotics, they had never seen a robot that didn't have a tool on the end of it. And so the idea of this thing being a sentient companion to us

was to really sort of accept it for what it is, to not adorn it with anything. And it kind of does have a face. It has that spherical thing with a whole pattern for the bolts. She doesn't see the world how we see it. She sees it through the ceiling. She has a bird's eye view of the world. But people still thought that she was seeing us

through the bolt pattern on her face. Let me still talk about the face, because you said it was important to have a face on your robot. No, yeah. I actually don't think we need a face. You don't need a face. You don't need a face to get a lot of personality from a robot. So this industrial robot that we see, I think we could all agree on where the face is. I've worked with robots that had a connect, you know, just a black box on their head,

and that was so clearly the face to people. I think we get a lot by not anthropomorphizing robots. So if you think about a lot of the intelligent robots that are starting to leave the lab to live in our world, these are all non-humanoid robots, drones, cars, some wheeled bots that are on our sidewalks. And those are things that like adding faces are expensive and they're just, it's like skeu-morphic

to what the actual machine is. It's almost deceptive in a way. So if we can find ways of just expressing their intentionality with their posture, their motion, and even the sound of their motors, then that's almost, those are the things that are native to that creature. And it sounds like we're saying something different

than what Justine was just up here saying, that faces and emotional expression is really important. But actually, we're saying the same thing. It's two sides of the same coin. You can get it with a face on a virtual agent, and you can get it with movement on a robot, and you don't necessarily have to give up that personality and that rapport. All right, let's go to...

See, I told you the kids would be first up with the microphone. Hi, go ahead. By what time do you believe these robots will be well incorporated to modern life. I mean... Is there a time for you shooting for any? I think we're living that time now. Yeah, it's 824.

So being in Pittsburgh, a hub of robotics in North America, we come across intelligent robots in our streets and in our public spaces and in our skies and in our parks where we might come across a drone. And these are all things that other people are putting into the world that affect our life in public. And that's something that's really interesting to be here in a time where these sentient machines are starting to share our world with us. Interesting. And they don't have faces either. Get used to it. It's not going to have a face.

What's the hardest part about teaching, Mimas? How about the learning curve there? A lot of the process with design is to make something quickly. and iterate, get it in front of people, and incorporate the feedback into making it more attentive or more responsive to them. So just getting willing participants to come in. I mean, we got, we were in a, I was doing this in, in Boston, in this beautiful warehouse space that Autodesco owns, and they had the glass facade. So we got lots of people putting their nose up to the glass. I'm like, come in, come play with my robot. I'm Ira Flater. This is Science Friday from WNYC Studios.

Talking about robotics here in Pittsburgh with two robotics experts. And how long did it take, you know, to train your robot or get to use to people? It takes a long time. So I don't, the robot that we used was a commercial product, which to program it, took some PhD students a year just about, and then to iterate on that design. I think actually something that's really important to mention is that we put our robots in front of people. We don't build robots in isolation. When you build robots for humans, it's incredibly important to actually have them work with people.

What about the fear people have about robots replacing them? I'm not afraid. I mean, I... You're also well positioned to not be afraid. I think that people are very good at some things, and robots are very good at some things. And robots are very bad at things people are very good at, like object recognition. Right, right. Or being able to reach a particular object through clutter.

We don't yet have robots that can reliably and successfully reach to the back of the fridge to pick up that can, you know, jar of mayonnaise or something like that. And we do that all the time as people. And so I think we're developing robots that are very powerful, that are very usable, but not particularly scary because they have these limitations. But, Madeline, your robots have already replaced people on assembly lines, things like that. How do you feel about the future of this? Yeah, I think that, I think it's a really valid concern.

something that I think that we should discuss more openly as a society part of what I think is at the heart of the fear is just a lack of agency. The idea that this technology is happening to us and we're not able to affect that change. Part of what I like

about doing this work outside of a research lab and inside cultural institutions like museums is that it starts to build eligibility for the affordances as well as the limitations of these machines. You sort of see what they can do and what they can't do well.

And you sort of give permission to have an opinion about it. See if I can get one question in from the balcony before we go. Yeah, go ahead. Why are the robots in black? Yeah, that's a great question. So for Mimas, Mimas is made out of metal, and I'm made out of squishy flesh. She's behind a glass box just to keep everyone safe

in case there's a wrong piece of code that I wrote. It's a good answer to a really important question. One of the reasons why robotics is such a hard feel to be in is that you can't be wrong. When a robot is in a home or when a robot is helping someone live a normal life, it's important that they're safe and that they're usable. Yeah, and actually something you said before is very useful for that. So robots that project their intentions that show us what they're about to do are very, very important because it helps keep people safer.

Yeah, people like a zoo animal, we're going up to them backing off, being afraid of. being afraid it was going to get them. So part of what we did with Mimas is we gave her a bit of attention span. She has an attention span. Yeah, she has an attention span. So she goes around and she picks the most interesting person in the crowd because she only has one face, and so she can only give her attention to one person at a time. So if you get boring, she goes on to someone else.

As soon as you stop moving, she's like losing, just like a zoo animal. Exactly, exactly like us. Thank you. This is fascinating thing. We wish we had more time. Thank you both for joining us. Thank you very much. My pleasure. Madeline Gannon is an artist, a roboticist at Envidia,

research fellow at Carnegie Mellon at the Frank Ratchez studio for Creative Inquiry. Henny Edmoni is a psychologist and assistant professor in the Robotics Institute at Carnegie Mellon and Director of the Human and Robot Partners Lab. Thank you both. Thank you. You can see videos and photos of my guest's robotics work on our website at ScienceFriiday.com slash robotics. Taking us to the break, our musical guests for the night,

Pittsburgh's very own townspeople. After the break, we'll talk about computers that compose their own music, and we'll let you hear a pop song written by AI. Stay with us. Still, I can barely see the light from all the bedrooms that I'll never dream in. I am the last king of the wild, the son of two for kids.

This is Science Friday. I'm Irafledo coming to you from the Carnegie Library Music Hall in Pittsburgh. Yes. And the theme of our show is no assembly required. And we've been talking about robots and AI, but we have to remember that behind every robot, as we've been talking about, is a piece of software that runs it. Every robot, AI technology we've been talking about, and you'll see tonight, is run on some sort of bits and things.

bytes, with one exception. That's what we're going to talk about now, a different kind of robotics. My next guest programs the stuff we wear, stuff we sit on, and now with software, she uses bacteria and chemistry. And if you wonder what that's about, that's what we're going to talk about now. She takes actual biological makeup of something, a chair, for example, and programs it to do something it didn't do before, like if you like IKEA stuff. off, she programs a chair to put itself together.

Yeah. As I say, we're talking about no assembly or computers required, and that's the theme tonight. And I want to introduce Lening Yao, assistant professor at the Human Computer Interaction Institute, director of the Morphing Matter Lab. I love that name. At Carnegie Mellon University, Dr. Yao, welcome to Science Friday. Thank you.

Just to explain further, you've invented something really completely different here, something different than we've been talking about. You don't use computers to create new designs. You use living things to shape things for you. Describe what you do for the audience. Sure. First of all, we do use computer. But the end outcome is a lot to do with the physical world.

And we do program materials. And as you mentioned, the name of the lab is called Morphing Matter Lab. And in a broad sense, we are considering intelligence are not embedded in hardware that are artificial. artificially built in a factory, but also intelligence could be embedded into materials and into our daily environment. And that's challenge the notion of robots. Robots could be the chair we're sitting on. Being robotics means it's being intelligent, know how to think, know how to sense,

know how to respond. And that's the concept we're working around. A lot of what you do is inspired by nature. Right. Yes. So about Pinecun, Pinecun is our unofficial lab symbol actually. So I grew up in Inner Mongolia in China, so we used to go out in the field to pick up pine cones after the rain because that's when the mushroom came out also. And so after the rain, the pine cones are actually fully closed, all the scales. And once you get it back home, put it on the balcony for a couple of hours, they actually open again. And this motion is reversible. So basically open and close for thousands of times. And in a way, a pine cone is a robot.

But it's our mother nature engineered smart robot that does not require electricity. So we got a lot of inspiration from it, and you will see it. We'll see it. Yeah, projects we're going to talk about. Let's first look at, I know people didn't think I was serious about a chair that could turn itself into a chair,

but you have actually created that. Yes, there was one project we recently worked on, actually. So the idea is you can print everything, 3D print, cheap thermoplastic indeed flat, and let it self-fold or self-assemble once you provide extra heat. So the vision is that for the IKEA chairs you bought, you don't need to manually assemble them anymore. You can just plug it into the wall or leave it in the oven to either bake it or just provide a little bit of electricity,

let itself fold into the shape you want. Wow, that folding chair was assembled itself. Yes, self-assembled. Actually, we brought one here. Let's bring it out. Let's bring out the chair. And look at it. Look at that.

Yeah, so as you can see, the leg part are not fully assembled. That's what we are currently working on. It should be everything self-assembled. But believe or not, I can sit on it. Do you want me to show? True. Look at that. Wow.

Do you have interest from stores like IKEA or anybody else who would like to create a chair like that? We are extremely interested in convincing furniture companies to work on this together with us. We are also talking about run some local workshops, maybe in Home Depot, so people can actually play with the robotic arms or 3D printers.

to print your flat furniture and try it out. You could print 3D print that at home, could you know? Yes, you can 3D print at home. We do it in the lab, and the lab just looks like home. A working space. A table. If you think that's something, now we're going to talk about some smart fabric.

I mentioned bacteria that you created, that has bacteria embedded into the fabric, and tell them about the magical properties that gives it. Sure, sure. That was a project. I started working on with a group of friends back when I was a PhD at MIT Media Lab. I was playing with the idea of, again, nature-powered actuators, and at that point, we were playing with bacterias. So it turns out the bacterias can be very responsive to the humidity

environment in the atmosphere. Basically, the bacteria will grow bigger when the humidity is higher and also shrink when the humidity is lower. And we figured the killer app will be a garment that can response to your sweat or body heat. Basically, when you are getting sweaty, the scales on the back of the garment will automatically open up to help you to get rid of the sweat and also cool your body down,

and then it will close once you feel cold. And it was a collaboration with New Balance because New Balance had the headquarters, half an hour drive from our campus. Wow, so actually the fabric starts out flat, and as you sweat, the sweat forces the bacteria to change their shape a little bit, and that opens up the flaps on the material,

so you're now feeling cooler. Have you tried this on? Yes, I was actually the first user who tried it on. Very brave of you. It was in the gym back still when I was in Boston. So what you felt is literally when you started to get sweaty, it's strange sensation that things are opening up and then little cool airstream started to travel from this hole

and enter the other hole. And then later on, so the dancer you saw in the video, he's from Boston Ballet Company. So he mentioned something also very interesting. He said it inspires me to create new dance moves because I felt like this strange, leaving things on my back and I need to be more expressive in the way I dance ballet. Yeah, different people have different sensations.

What does it feel like? Does it feel like regular garment when you're wearing it? Yeah, yeah. Actually, it's just a normal garment. And that's where actually new balance kick in. So most of the, like the sports fabric and also the way to do seamlessly, no sewing line, you do it through a thermal bonding. And all those techniques were shared by new balance to just make it comfortable to wear. So the bacteria are doing the reshaping and opening of the flaps. Do they remain alive in the fabric?

So an interesting point. On the science aspect, you do not need it to be alive because we are not using the metabolic function of the bacteria. It is the protein and DNA components within the bacteria that are responding to humidity. But as a matter of fact, the special bacteria we're using is called nato-bacteria. Maybe some of the fox know Japanese. Japanese eat it. It's their breakfast. It's fermented soil bean. And they turn into

zombie once the nutrition is not enough. The scientific term would be endosbore. Basically doesn't grow, it doesn't replicate, but it's kind of, yeah, freeze there. If you give it more nutrition, it can become alive again. So we were actually thinking of the next step. We wanted to see if your sweat can feed the bacteria. And can make it glow. For example, you are running in the dark night, your shoe will open up and also glow. Wow. You got them at zombie, I think, is when I got very interested. Why? I mean, but I would imagine if it's bacteria and it's in the fabric,

if I throw it in the laundry, it's going to wash it out. Is that a problem that you have to overcome? So it wasn't a serious problem if you were trying to just gently wash it. it, but you are right, if you're putting in the washing machine, they are gone. But then bacteria are actually really good at attaching at things because of the glue on the surface, and that's what they do for survival. We were also a little bit leveraging that. Could they be, in this case, could there be unintended consequences of the fabrics and the things that you're working on? Your chair, that you might not have thought about. Yeah, I mean, even the last one, the bacteria, people have been writing.

on the social network, they were like, oh, through my life, my parents told me to get rid of bacteria. Now you guys are encouraging us to put them on. And also when I say we want to program the bacteria to be fed by sweat and glow, we are also manipulating life in a certain sense. Sometimes it works, sometimes it doesn't. So those things are often coexist with the advance of technology as well. But actually, as a human computer interaction researcher, part of our mission is to leverage the benefit and trying to use technical invention and also design intervention to kind of mediate some of the risks. It's a never-ending conversation. I'd like to talk more. Well, let's talk more about some of the other really interesting projects you're

working on. And that's something really fascinating about food. Pasta, you have found a way to have pasta form into its own shapes. Yes, so can we imagine? Starts how flat like that. the chair. Yes, can we imagine? So food is robotics nowadays as well. So this is a project we worked earlier at MIT in collaboration with Target. But nowadays we brought it to CMU also in collaboration with Barilla. Barilla is one of the biggest Italian pasta company.

So we were trying to make autonomous self-assembly food. So think about your macaroni pasta. pasta actually waste 60% of the packaging space compared to the flat pasta, such as thin hair spaghetti. So we were convincing the pasta manufacturers that you can make all the food, all the pasta flat and then trigger them to sell fold into different shapes on the flat. How do you get them to fold? You're starting flat. They're folding up into different kinds of pasta. How do you get them to know which shape they want to be? when they grow up.

Yeah. So there are two parts. So first part is the material science part. Actually, everybody knows here when you cook pasta, they grow fatter. They grow fatter because of the hydration, meaning swelling. We are just kind of get geeky out of it. So basically we try to study different flowers, different components within flour, and compare their differential swelling rate and recompose the pasta late.

the pasta layer by layer, if they swell differently, as you can see from our microscope image, they started to take on a 3D shape as you cook them. Yes. So different parts of the pasta, but you make it out of different things and it swells up in different rates. Yes. So it'll start to twist and bend. Yes, thanks for explaining. It's in another way.

As George Kirstenger said, this is what I do. But then there's also a little... also why you can achieve so many different shapes, where you can replicate a classic Italian pasta shapes, that part is actually the computer science part. So we do program, but we program physical materials through its composition and structure, and those defines eventually where it swells and where it doesn't. In the grand scheme of then of robotics and self-forming objects, Where do you think this is going to eventually head? Where would you like to see the whole industry go?

You mean the whole industry of robots? You're working on. What I'm working on? Yeah. So I just attended a workshop with actually a group of robotic professors, and the workshop's name is called robotic materials. So at least our lab, the mission, is that we wanted to bring the notion of robotics into our daily life.

And again, I wanted to just sorry, say, being robotics means being autonomous, know how to sense, how to response, and how to think. And our materials in the physical environment can do so. Your clothes can do so, the speakers can do so. You can make anything virtually, you think. You can make a cup of coffee, you can come up here by itself?

Yeah, so you can make a coffee mug turn into a wine glass depending on what you pour in. That's one of the dreams. I've seen that on Star Trek somewhere. That's very fascinating. Thank you very much for taking time to be with us today. Lening Now's assistant professor at the Human Computer Interaction Institute, Director of the Morphing Matter Lab at Carnegie Mellon University. This is Science Friday. I'm Ira Flato coming to you from the Carnegie Library Music Hall in Pittsburgh.

Yes, we have talked a lot tonight about how robots can partner with us at work and how artificial intelligence can build trust in its human counterparts. But what if we unleash some of these machine learning smarts on the creative arts, train computers to write their own music, or teach instruments to play themselves with a few tricks from robotics? My next guest are the music teachers of the robotic world, musician engineers, who are working to make our robotic future a little more melodic. Roger Dannenberg is a professor of computer science, art and music at Carnegie Mellon University.

Welcome to Science Friday. Thank you. Eric Singer is a musician and a roboticist based here in Pittsburgh. Welcome to you too. Thank you. Roger, you've also built instruments that play themselves. We have a clip here of a robotic bagpipe player you built called McBlaher. Yes.

Let's listen to it. That's great. That's wonderful. Thank you. Thank you. It reminds me of a computerized player, piano except you've done it with the bagpipes. Yeah, that's...

They made a player of bagpipes. What's the brains behind an instrument like that? Well, there are a few different parts, but mainly there's an air compressor and regulator that is needed to drive air into the bagpipes. And then there's a set of robotic fingers. They're operated electromagnetically to pull little pads down to the tone holes. And they have to be very fast because bagpipers play very fast. play very fast music with trills.

And so that was part of the challenge. And then the fingers themselves are controlled by a little micro-computer that's receiving commands from a sort of a control computer. That's playing a song that's in the computer. Right, right. The songs are stored in the computer. Wow. And I should, I want to credit Garth Zeglin and Ben Brown

from the Robotics Institute at Carnegie Mellon, who were co-creators of this, McBlair. McBlair. I tried the Scottish Brog, but it didn't come out too well when I said. Eric, you've also designed a whole robotic orchestras. Several. In the Lido Nightclubs in Paris, right? The Lido Nightclub in Paris has an orchestra that's about 40 or 50 instruments.

There's about 250 mechanisms going there. And we also had one that went on tour with jazz, a guitarist, Pat Mathini. Let's see that clip and then we'll listen to what it's going on in that one. The only thing that Pat was playing in that was the guitar, right? That's right. They were all, and they was improvising and they were able to follow all those computers. Some of it he's following them and some of it they're following him. I'm just aghast that how that could happen.

How do the musical instruments know how to follow him or what we anticipate, or how to riff on their own? sophisticated software that can do a number of things. One, they can listen the way we have computer vision that can look at things. We have computer audition or hearing that can listen to the sounds, musical sounds coming from one instrument and interpret them and then make up material based on that in whatever ways we as software composers want them to do that. And Roger, I know there is a long history of people trying to automate things. Is there not?

I mean, human fascination with automation goes way back. Right, right. Automation goes back along hundreds, thousands of years. And I think that people have always been fascinated with it. So that's one thing I like to point out is that people doing robotics today are not really doing something completely new or completely challenging and turning the world upside down. upside down because, I mean, for example, there were three brothers in the ninth century from Baghdad that published this incredible book. It's called The Book of Ingenious Devices, and they had about 100 mechanical inventions,

including a robotic flute player that included digital storage. It was all done mechanically, but it could actually, I mean, there were pictures in the book of how to store songs on a drum and reproduce those. with a flute player. It's interesting about the history of robotics and music. For years have been looking at the history of Hetty Lamar. And of course you can't talk about Hayley Lamar, who had a great mind for mechanics and invention without bumping into George Antille.

Yes, he wrote a piece called Ballet Mechanique intended to be performed on 16 player pianos and a human percussion orchestra. Unfortunately, the company that told him they could make 16 pianos play in sync, actually couldn't. Fast forward to now, you have digital player pianos where you can use a standard interface called MIDI that almost all musicians know how to use, that will actually physically play them just the way a player piano would. What is the advantage of having a machine play music? I mean, we go to concerts because we watch the people, they have energy, they have life in them, they have expression. Explain to me,

the fascination with just having a machine play the music? It looks cool, first of all. Everyone gets fascinated by it. That's fair. I mean, in almost everything that I've done, there have been human musicians collaborating with robotic musicians. When we talk about software, as Roger writes and some that I've written, that actually composes or improvises music,

it's really us that's creating the music, but we're doing it through software. Roger, you agree. I'd like to say also musicians and composers are always pushing boundaries. And going back to history, for example, we have pianos because there are things you can play on a piano that you couldn't possibly sing.

So there's an incredible technology in the modern piano to enable a new kind of music that led to all kinds of developments in harmony and style. And so I think we use computers and robots and anything we can get our hands on technologically in music to explore new territory. Let's go up to the balcony for a question. Yes, yeah, with your hand up. So has there been any backlash from the classical musician slash composer community regarding the automation of composition?

Because if you can create artificial intelligence to compose more music, is this kind of like the same situation where it could put modern day composers out of a job? Yeah, I think, first of all, musicians tend to be very open to exploring new things and new ideas. And I work with a lot of classical musicians, jazz musicians. I'm a musician myself. And this, you know, new territory, it's just like if you write music in a new style or ask the musician to use some different playing techniques, asking a musician to play with a robot or play some computer-generated music is just another kind of music expression. So I don't see any problems there.

All right. Equal time to this side of the balcony. How many trials and errors have you had? Many, many, many more errors and trials than successes. One of the great things when you're developing software, software for computers to play music is that sometimes you make mistakes and something comes out that you never imagined and it could actually sound really good. I've spent hours trying to understand mistakes that I made and codify them so that I could reproduce them because they were so great.

I spent hours fixing mistakes. But it's good that even, you know, youngsters understand that it takes trial and science is not about always, you know, getting it right the first, time. Failure is a big option in science. It's very experimental. And regarding the idea of putting musicians or composers out of business, I get asked that question a lot, but I've never woken up, for instance, and found out that the robots have started their own band and composed an album. Humans, like humans used, the piano, the violin synthesizers, which people were very afraid of to begin with. All of this technology, to advanced arts. Visual artists use Photoshop.

They use generative visual techniques. Why not use generative audio techniques? There's a human always making decisions in the end, like the computer composed a bad song. I'm not going to put it on my album. Let's talk about machine learning and music, whether artificial intelligence can be creative, Roger. I understand you have a computer that can write its own music and lyrics and all kinds of stuff like that. How does that work? Yes. Well, I've been working on a program to compose pop songs based on some previous work that others have done, and there's quite a community around the world now looking at questions like this. The way my program works, and we're going to hear some output in a while,

I hope. It does things in layers, so it begins thinking about the structure of a song, and it knows a little about possible structures, and it basically just picks one. And then it tries to fill in that structure with some rhythmic accents. It doesn't know what it's going to put there, but it sort of consistently follows the structure. As you wish, we're going to listen to one right now. Yeah, okay. Let's listen. Here's a clip, a special song for tonight's show, a clip of what the computer wrote. Let's play that now.

Oh, I got to tell you. Give it up. That ain't gonna make it to the top ten. I give it two elevators. That was good elevator music, you know, but... Hang on because we wanted to see what it would sound like if real human people, a real human band played that computer composition, and our band tonight, townspeople,

They said they would be up to the challenge. Come on out. Bring them back out here. What happens if you take a computer synthesized music and you combine it with the talents that people have? Maybe it will sound a little bit different, a little more creative, a little more fun. So are you guys, you guys ready to give it a shot? Let's hear what the talents people can do with that little tune. little tune. I'm I replayed it. This is Science Friday from WNIC Studios.

Now Alex, you knew we were asking to play this and then what was your first reaction when you heard that other version of it? Well, we were, you know, we were a little intimidated. We thought we were safe as songwriters from the revolution, but no, I really liked, you know, the lyrics especially. They're no more absurd than your average pop song. Well, thank you all for, you know, for, you know, for the lyrics. bringing that song to life, townspeople, people. Thank you for writing it. You're welcome.

Hey, I want to say that the lyrics was kind of a recent edition that was really by request of townspeople, so I want to thank them for motivating some new research. Well, that brings up a, you know, I'm always interested in the business aspects of science and technology. Now, if a computer wrote that music, who owns the copyright to them? I do.

So the programmer owns the copyright and can't the computer sue you, take you to court? I've never talked to a lawyer about this. So it's a really interesting question. And it gets much more interesting if you sell the program and then someone else uses the program to produce it. And we have similar problems with recordings, with samples. You take a sample and load it into a synthesizer and play it on your song. who owns, you know, so it's... To me, Roger wrote the song, he just used software to do it.

Oh, that's... Ooh, I like that answer. Fascinating. Thank you both. Gentlemen, Roger Dannberg, Professor of Computer Science, Art and Music at Carnegie Mellon, and Eric Singer, a musician and roboticist based right here in Pittsburgh. Thank you both. Thank you very much. That's about all the time we have. Our heartfelt thanks to 90.5 WESA for hosting us and to WESA's

Elizabeth Bazely, Nick Wright, Dorothy Frank, John Sutton, and Terry O'Reilly, and the University of Pittsburgh, All of Us Research Program, and the Carnegie Science Center for the Robotics, and to Mike Stallone and everyone at the Carnegie Library Music Hall for making this wonderful evening possible. And thanks to all our Science Friday staff, it takes a lot of people behind the scenes to run this ship. Oh, and let's give one last round of applause to our amazing musical guest townspeople

who are going to play us out tonight. Thank you for coming. Drive safely. Good night, everybody.