Planet Money - AI Podcast 2.0: The host in the machine

Episode Date: May 31, 2023

In Part 1 of this series, AI proved that it could use real research and real interviews to write an original script for an episode of Planet Money. Our next task was to teach the computer how to sound like us. How to read that script aloud like a Planet Money host.On today's show, we explore the world of AI-generated voices, which have become so lifelike in recent years that they can credibly imitate specific people. To test the limits of the technology, we attempt to create our own synthetic voice by training a computer on recordings of former Planet Money host Robert Smith. Then we introduce synthetic Robert to his very human namesake.There are a lot of ethical, and economic, questions raised by a technology that can duplicate anyone's voice. To help us make sense of it all, we seek the advice of an artist who has embraced AI voice clones: the musician Grimes.This episode was produced by Emma Peaslee and Willa Rubin, with help from Sam Yellowhorse Kesler. It was edited by Keith Romer and fact-checked by Sierra Juarez. Engineering by James Willetts. Jess Jiang is our acting executive producer. We built a Planet Money AI chat bot. Help us test it out: Planetmoneybot.com.Help support Planet Money and get bonus episodes by subscribing to Planet Money+ in Apple Podcasts or at plus.npr.org/planetmoney.Learn more about sponsor message choices: podcastchoices.com/adchoicesNPR Privacy Policy

Just a heads up, this is episode two of a three-part series exploring whether artificial intelligence can make a Planet Money episode. Today's episode does make passing mention of a couple of technologies from Apple and Google. You should know that both of those companies are financial supporters of National Public Radio. Here is today's show. This is Planet Money from NPR. This is Planet Money from NPR. Here at Planet Money, we have been testing these new artificial intelligence technologies to see how much of our jobs they can actually do. Last episode, we used tools like ChatGPT in some moderately complicated ways to write an entire Planet Money episode. And that episode is finished.

We have a script. We are excited for you to hear that podcast. But if we are really going to take seriously the question of, is the AI going to replace us? We wanted to go all the way. We needed the computers to also replace our voice to narrate this episode. And if we were going to have the computer do the talking, we might as well go big. Get the most Planet Money e-voice of all the Planet Money voices.

I speak, of course, of... Hello and welcome to Planet Money. I'm Robert Smith. Robert Smith, Planet Money's longest-running host, even was the show's editor for a while. But a couple years ago, he finally hung up the old microphone Smith, Planet Money's longest running host, even was the show's editor for a while. But a couple of years ago, he finally hung up the old microphone and left the show as a full time host. Robert left behind a legacy, but also lots and lots and lots of recordings of his voice. Hello, I'm minor podcast celebrity Robert Smith.

And here at Planet Money. In order to save the country, they're going to have to work together to restructure the tax code. Who is this poop cartel that you speak of? Ahoy there! Welcome aboard the good ship inflation. And just for fun, we did try to hack together our own version of a knockoff Robert Smith using those old recordings. We were just, you know, cutting and pasting those words together. Yeah. We even gave him a little catchphrase and, uh, here, here, give it a listen. I'm knockoff Robert and I'm not here to make friends.

I'm here to make history. Uh, yeah. Kind of hard to understand there. He's saying I'm knockoff Robert. I'm not here to make friends. I'm here to make history. I'm knockoff Robert, and I'm not here to make friends. I'm here to make history. Yeah, okay. It's not very good. And clearly, there are better ways to do that now, because part of the AI boom has been the introduction of these surprisingly realistic AI voices. Yeah, somebody made a completely synthesized conversation between podcast host Joe Rogan and Steve Jobs, who is dead. How's it going? Come on, tell me about Jobs. Yeah, it's great to be on the show. Your audience is just so different from your normal Apple users,

and that's a good thing. And then there was a song featuring an AI-generated version of Drake and The Weeknd. I came in with my ex like Celine, not a flex. So yeah, AI Drake, AI Joe Rogan. Clearly this is possible now. So today's mission? Go and somehow build the Planet Money version of this. AI Robert Smith.

Hello and welcome to Planet Money. I'm Kenny Malone. And I'm Jeff Glow. And today, the next step in our quest to try and automate our jobs at Planet Money, can a computer believably serve as the voice of planet money? There has clearly been some enormous leap in making computer voices. So what happened exactly? How did we go from like bad GPS voice in half a mile turn left to that incredible Drake thing?

We will build and introduce you to a fully synthetic Robert Smith. We will build and introduce you to a fully synthetic Robert Smith. And we'll also check in on another synthetic voice clone of an even more famous celebrity, Grimes. Wait, wait, who is Grimes? Sorry, knockoff Robert. Grimes is like a musician slash cyborg slash like demigod. It's complicated. Ask your kids or whatever. Okay. Stay with us. Hello. Hello. This is Martin and Matt Hawking. Thank you for having us. This is Martin Ramirez and Matt Hawking. They're from a company called Well Said Labs.

It's a Seattle tech startup that makes what they call voice avatars. Okay, so we are interested in taking our beloved colleague, Robert Smith, and somehow making a version of him that we can still have in our wonderful episodes occasionally. Is that something that you guys can do? As long as Robert is in, Well Said will be well-equipped to create a synthetic replica, if you will, of Robert's voice.

Synthetic Robert. Love it. Synthetic Robert, an upgrade from knockoff Robert. Now, Matt and Martine, give us a quick little history of people trying to get computers to talk like people. I think when we originally started down this path, text-to-speech had primarily been very robotic, very monotone, and had been produced in a concatenative way. What is the word you're saying? Concatenative? Concatenative. In this case, it means taping together a bunch of pre-recorded bits of voice. So think about the way that we built that funny knockoff Robert earlier.

I'm knockoff Robert, and I'm not here to make friends. I'm here to make history. Yeah, well, that is a bad version of how synthetic voices used to get made, according to Matt and Martine. A much better version is the original Siri voice that was recorded back in 2005. And that voice actor said that they made her go into a studio and she had to read these weird phrases like cow hoist in the tug hut today or say shift fresh issue today. Well done, Jeff. Well done. The reason she had to say those weird things for like 80 hours, by the way, was so that in the end, Apple could chop up all the recordings and have like a drawer filled with. So that is the old way. The way that Matt and Martine were going to build synthetic Robert was apparently way different.

We're not going to build a drawer of Robert making all of these sounds. Yeah, he's not chained to a microphone saying millions and hundreds of millions of permutations of his scripts. Yeah, as far as I know, he's like cross-country skiing right now. And that there is what we inevitably have created is the ability for Robert to be off cross-country skiing. Because the modern voice approach by well-said and an increasing number of other competitors in this space is quite different from the old thing. For example, if we're going to go ahead and do this Robert thing, Martine and Matt explained that we're going to send over some recordings made by Robert Smith during his time at Planet Money.

And then we'll also send over the exact transcripts for those recordings. And then essentially what's going to happen is that a computer is going to look at a chunk of that written transcript and it's going to try to guess how Robert might say those words. So whether that be the pitch, the pausing, the intonation, the emphasis, all of those types of things have certain metrics which are measured. And I love the perspective of let's be wrong multiple times as quickly as possible. And it's going to sound sort of kind of, but maybe not really Robert.

Wrongbert. Wrongbert. Yes, there you go. And this synthetic voice is going to be wrongbert for a long time. Martine says it's going to take millions of training cycles. But in the end, we should end up with a computer program that can look at any sentence and do a really good job guessing how Robert Smith would say that sentence. And by the way, if it took 80 hours of recordings to make Siri using that old concatenative approach, this new approach, it requires way less audio. We've kind of come to around 45 minutes to a couple of hours worth of audio to get to a point

where we can create a very realistic likeness to someone's voice. A full voice clone from so little audio. And, you know, to be clear, it's not just well said here. Like this technology is spreading really fast. Yeah, there are lots of companies with their own versions of this voice cloning stuff. Google has one. There's another company called Eleven Labs. Apple says it's about to get in the game. Plus, there are open source voice cloning tools without any real restrictions or oversights.

And anyone can use them. And of course, when you start to consider the implications of all of this, it is really terrifying in some very obvious ways. I mean, imagine how bad it would be if there were a synthetic Joe Biden voice announcing nuclear war or something, right? Yeah. So companies like Wellstead, they are well aware of these fears. And so Wellstead explained to us that if we were going to work with them for this project, there were going to be lots of terms and conditions. For starters, Matt and Martine told us that they wouldn't even consider making a synthetic Robert Smith until they had heard from the real organic Robert Smith and had his permission. They also told us that they will be able to monitor every word we make synthetic Robert say. As soon as anything looks like it's not in line with our values, we're very quick to put an end to that.

And maybe the biggest term and condition of all, as soon as we were done with this project, Synthetic Robert would be shut down. He could narrate our AI-generated episode, and then he will be functionally destroyed, never to be used again. Poor Synthetic Robert. But we agreed to all these terms. We got Robert Smith's permission.

We went into the old NPR Planet Money archives, and we pulled a lot of Robert Smith recordings. And I'm Robert Smith. And I'm Robert Smith. And I'm Robert Smith. And I'm Robert Smith. And I'm Robert Smith. He said other things in these recordings too. But yes, we sent all of the recordings over to Well Said, and we waited. After the break, synthetic Robert Smith makes his world premiere. Hello, it's Amanda Aronchik and I'm here to say thank you to our Planet Money Plus supporters.

You help make our work possible. And one really important part of that work is fact-checking. A lot of times Planet Money hosts will talk to a source for two or three hours. My job is to go in and make sure that what we have reflects the entirety of the interview. Meet our resident fact-checker and go inside our fact- fact checking process in our latest bonus episode out now for our planet money plus supporters if that's not you it could be learn more at plus.npr.org while we were waiting for synthetic robert to train and i know i know this isn't how it works but i'm definitely picturing it like rocky running upstairs in philadelphia but yes uh we were waiting for that to happen, we started to think more and more about some of

the other implications for this voice cloning technology. And, you know, not just the, oh no, what happens if someone makes fake Joe Biden, but also what happens economically if suddenly voice clones of the thousand most famous people are commercially available. Yeah, what does it look like when there's suddenly a synthetic Robert Smith who never gets tired, who can narrate an infinite number of Planet Money episodes? Yeah, or like what happens if, I don't know, Tom Hanks suddenly makes a cheap cloned version of his voice available for all commercials and all audio books and all documentary films. And weirdly, we can already get a peek into this bizarro future. It's starting to play out right now with Grimes.

Grimes. Yeah, the experimental sci-fi pop musician from the future. I mean, she is here in the present and is writing music, but she seems like she's from the future. But yes, here's some of her music. Grimes has a really distinctive voice, like whispery and ethereal. And in the middle of our AI project, Grimes made this kind of wild announcement on Twitter. She said basically, hey, from now on, anyone is allowed to copy my voice using AI and like put me on their songs. Just split the royalties with me if you do. And Grimes

even released her own AI voice cloning tool that anybody can use to turn into Grimes. And so just allow us to show you how this works. It is stupidly easy. So let's say you have a song that you want Grimes AI to sing. Oh, I do. It is a weird version of Row, Row, Row Your Boat where every word is off by one note. Yeah, that sounds okay. I cannot even imagine what that sounds like. Oh, it's weird. Okay, so you are going to record a version of the song. I will. Here we go. A dream row dream row row row your bow gently down the stream merrily merrily merrily life is like a dream row row is that enough do you have do you need more no no no that is that is plenty that is plenty to work with so yeah so i have a recording of you singing that song

that horrifying song uh literally all i have to do is i hit upload do you are you ready to hear yourself as grimes oh my god God, I'm so ready. Okay, we're going to throw a little beat under it so it really sounds like a Grimes song. Okay, here we go. A dream row, row, row your boat gently down. A stream merrily, merrily, merrily life is like. Oh my God.

A dream row, row, row your boat gently down. So using this technology, Grimes is essentially unleashing her voice to go out and make music and earn money on her behalf without her having to be there at all. And so, of course, we had to talk to Grimes about all of this. I'm sorry, Planet Money, but I'm unavailable for an interview right now. No, actually, sorry. That was Kenny singing into the machine again. However, we did manage to talk to Grimes' like co-conspirator slash collaborator on all this technology stuff. Hi, I'm Dawouda Leonard.

I'm the co-founder and CEO of CreateSafe. And additionally, I am Grime's manager. Do you know how many songs are out there with Grime's AI? So we've had around 300 songs distributed or uploaded to the platform to be distributed. So it's 300 songs uploaded through the platform in what time period? What's today, the 18th? Yeah. So 18 days?

I just want to stop and reflect on how bonkers that is. Because in just 18 days, because of AI Grimes, effectively 300 new Grimes songs were born into the world. And the real Grimes didn't have to do a thing. Like that is a productivity gain. I mean, here, just listen to some of these songs. So yeah, everything you just heard, that was AI Grimes. And now all of those songs, they're just floating around out there. And if any of them take off and make money, boom, Grimes gets a piece of that. And Dauda wanted to tell us about

this one specific song somebody had made with this AI stuff. This song right here. So this was created by an artist called Keto. And Dauda told us that the real Grimes heard it and liked it so much that she is now going to add her real voice to the song. There's the original Keto version. And now there's Grimes AI. With Grimes AI. And then there will be like Grimes and Grimes AI and Keto. Oh, Grimes AI stays on it. Grimes AI is going to stay on it, right?

Like there's going to be like some of her vocals still on it. The splits will be interesting, right? Because like Grimes AI is a 50-50 split and then plus a new Grimes, which is like another 50-50 split. Yeah, but Grimes isn't going to try to like, nah, I don't think that's not how Grimes rolls. No, no.

Because if you ever get a chance to talk to Grimes about all of this stuff, like it's clearly not really about the money. And we know this because we did ultimately get a chance to talk to Grimes about all this stuff. And this is gonna take a little bit of explaining. So go ahead, Jeff. Yeah. Okay.

So after our interview with her manager, right as we were wrapping up this episode, we finally heard from Grimes herself. She agreed to answer some of our questions via email. And then we got those emailed answers as forwarded to us by her manager. But, you know, like Grimes is a self-described self-replicating pop star and, quote, basically considers herself a cyborg. self-replicating pop star and, quote, basically considers herself a cyborg. So in a planet money first, we believe, Grimes has agreed to speak her written answers via synthetic voice clone. That's where we're at right now, Jeff. So we begin this strange interview with the most obvious question. We asked Grimes, why did you decide to give the entire internet control of your voice?

We've been trying to do this for like five years, but the tech just wasn't there before. So we moved on. And when the Drake weekend stuff went viral, I realized this was pretty accessible technology. And to be honest, I didn't even think about it or ask my team. I just tweeted that people could use my voice and it went viral. it or ask my team. I just tweeted that people could use my voice and it went viral. Again, that is an AI Grimes voice clone speaking aloud to the answers that the real Grimes emailed to us, Planet Money. Yeah. Our next question for Grimes, what does having an AI voice clone version of yourself allow you to do that you couldn't do before?

Open the creative process to all humans. Grimes to me is art, not music. And while getting credit for my work is cool and used to matter to me a lot, accessing the human hive mind in a way that wasn't achievable even a few months ago feels like the most exciting thing. Now, there was this one big thing that we especially wanted to talk about with Grimes, a sort of economic problem that Jeff and I had been worrying about as we were making our own synthetic voice clone of a less famous, but still famous-ish person, Robert Smith. And that economic problem that we were worried about is what economists sometimes call the

superstar effect. So yeah, the idea is that sometimes new technologies can vastly increase inequality. So think about the printing press or radio or television, these tools that let one person reach massive audiences, beam themselves into every single household. They can create these superstar economies where a tiny handful of actors or artists, they become really popular, make a lot of money, and then all the other actors and artists, they get pennies. And so our question for Grimes was like, why won't this also just happen with his voice cloning stuff? Like, doesn't it just let the already famous people just replicate and become bigger? And Grimes, you know, acknowledged to us that, yeah, she does have concerns about those sorts of economic phenomena. Yes, absolutely. I think this is also a debate to be had.

I'm hoping smaller, mid-sized artists like me can workshop these phenomena a bit. But she also said, look, it is also possible that releasing her voice to the world could create opportunities for up and coming musicians. Like getting your work to be noticed is really hard. But now if you're, you know, a young songwriter or producer or whatever, you can kind of borrow Grimes's stardom to get your own name out there. I run into absurdly creative humans all the time, but not a lot of people get to be artists. A lot of luck is involved in that. It's hard to build a fan base and it's hard to get your work in front of the public. So if there's ways to reduce these algorithmic barriers by letting people inhabit my being,

then I think we're moving in a direction I really like. It's a great point. Again, being spoken into the world by the synthetic voice clone of a cyborg self-replicating pop star, a disclosure that we need to keep making. It's so weird. And you know, the big theme that we've been wrestling with in this podcast series that we're making is, what do these new AI tools mean for our jobs, you know, specifically as journalists who work in radio. And this strange cyborg interview with Grimes, it was incredible. It was alarming. It was all of these things because we talk a lot in radio about the intimacy of the human voice. And I don't know, it totally felt like at moments we had spoken to the real Grimes,

that we had this like real personal connection with her. Oh, for sure. Which of course we had not. We'd had no personal connection whatsoever. We'd gotten an email. That was it. It's bizarre. And, you know, look, it's one kind of bizarre, I think, to talk with the clone of some far

away pop star. One kind of bizarre, I think, to talk with the clone of some faraway pop star. But what was still left for us, Jeff, was to meet the clone of our beloved colleague, Robert Smith. And after a couple weeks, we got the call. The brand new synthetic Robert Smith was ready. The company we worked with to make synthetic Robert, well said, they wanted to show us the final product. And so we hopped on a call with Ryan Johnson. She was in charge of creating and training Little Synthetic Robert. Hey, Kenny, how's it going?

This is a little bit like Christmas in the spring. It really is. Well, if this is Christmas, I feel like the excited mom who knows the gift that's coming. And I wrapped it late last night. Okay. Now, one quick thing we have to mention before we unveil Synthetic Robert. Creating one of these voice avatars, as Wellsaid calls them, it's not cheap. They told us that what we're doing with Synthetic Robert would typically cost about

$20,000 to $30,000. But in this case, Wellsaid did do this for free for Planet Money. And okay, I think with that said, let's get into it. Ryan Johnson from Wellsaid began by explaining all the work that had gone into creating this weird Robert Smith. Let me start, let me, so we talked a little bit a few weeks ago just around creating a predictive model that can generate audio like this, right? Ryan gave us a quick refresher on the basics of how she made Synthetic Robert. Yeah, so she had a computer look at some written text and then guess how Robert Smith would say those words. Then the computer checked its guess against a real recording of Robert Smith, saw how far off it was, tweaked its approach, tried it again, over and over and over, hundreds of thousands of times. So yeah, Ryan said all of that training was now done.

And she was actually going to let us hear how the voice progressed over time. And so she starts by playing what Synthetic Robert sounded like at the very, very beginning of the process. So if you're ready to hear step zero. This is guess one. This is essentially nothing. Here we go. I personally believe that F-bomb is for expectancy and it's because the machine is.

It sounds like a swarm of bees are coming out of Robert's mouth. Yes, that's right. It's very noisy. It's gibberish. In fact. It sounds like he's trapped in the Matrix. Yeah, it does. Okay, so what's the next one? How many steps?

100,000 steps. 100,000 steps. Okay. Here we go. 100,000 steps, Robert. Okay, here we go. I'm synthetic Robert, and I'm not here to make friends. I'm here to make history. What? That is pretty good. Are you excited?

It is like Robert in a water chamber or something. But that's Robert Smith. And here's the final final. I'm synthetic Robert, and I'm not here to make friends. I'm here to make history. Holy crap. Oh, my God. That last part, it sounded...

Oh, my God. That's bonkers. I don't even... Jeff, what? I mean, this is amazing. How do other people react when you show them this? Yeah. I think similarly.

Yeah. Now, Martin Ramirez, also from the Well Said Company, was also watching our reaction on this call. And he jumped in to say that our dropped jaws and wide eyes and general speechlessness here like yeah that kind of is the reaction that this always gets and i think kenny and jeff i think this illustrates um the complexity not only from a technical perspective and your reaction you feel you're listening to robert it's it does i'm sure you see this martin though it is like we're aghast and like also like And your reaction, you feel you're listening to Robert. It does.

I'm sure you see this, Martine, though. It is like we're aghast and also like, oh, God, what have we done? I was fully ready to ask you to have Robert say the stupidest stuff in the world. But I do suddenly feel like a steward of this as opposed to a wacky prankster. And knowing that, right after that moment of wow comes a realization of this is not a toy. Right. So it is indeed a very heavy sense of responsibility

in making sure that we are properly ushering this experience and putting these capabilities as well as we can when we take them to market. Yeah. And this is part of why our little beautiful synthetic Robert Smith is not long for this world. As we mentioned, the deal we made with Well Said, and of course with the real Robert Smith, is that synthetic Robert will exist only for this project that we're working on right here. And then that's it. Plug pulled. But of course, before any of that,

there was one more person who needed to hear Synthetic Robert. We set up a Zoom with the real Robert Smith. Hello, Kenny Malone and Jeff. How's it going? It's going great. Hey, wait. Wait, somebody who's what's happening here? Somebody else is joining here. What is up, organic life forms? S synthetic robert has entered the chat what the hell i just want to say robert you or should i say we have a beautiful voice robert's hands are on his face Robert is screaming

He's massaging his forehead Are you okay? That's better than I thought It's got the pauses and the corrections Oh my Oh play me more Play me more of me Okay well what do you want to say?

What do you want to say? Well I mean like Like the very the very basic like hello and welcome to planet money like i've said that a million times there we go hello welcome to planet money i'm robert smith not bad a little little little uptone on the smith you got notes huh i do have notes i'd like to have a talk with myself here's an interesting weird thing is if I run it again it will not do

the exact same thing because it's guessing probabilistically each time right here hello and welcome to Planet Money I'm synthetic Robert Smith that is pretty good

that is pretty good because that one's it's tossed off but you know there's a little bit of confidence behind it yeah like I've done this before

good good and I guess I I guess I should be um a little bit freaked out but like the first place my mind goes uh-huh is this could allow me to be lazier than i am which is i like i like narrating and enjoy narrating, but I hate it when like the producer calls up and be like, Robert, uh, you know, it turns out Minneapolis is not the capital of Minnesota. Right. Can you do a retract? And I'm like, oh, I'm already, I'm already relaxing.

Um, so it would be great to have this, to just be like, yeah, have Syntho do it. it would be great to have this to just be like, yeah, have Syntho do it. Basically, like my mind immediately goes to how do I do the fun stuff and make Robot Me do the chores? And you know, this is sort of exactly the use case that people talk about with this technology. Like you can imagine if you're a celebrity author, you know, just have your voice clone, read your audio book. Why waste your time? Yeah. Or if you're a voice actor who does TV ads, just send in your voice clone while you lounge on the beach. But in the case of Synthetic Robert, you will recall he was created for one thing and one thing alone, narrating the fully AI written episode of Planet Money that we have been making here in this series.

And next time in the third and final episode of our series, the world premiere of that episode, written by AI, narrated by synthetic Robert Smith. And possibly the last time humans will be involved with a Planet Money episode. You'll have to stay tuned to find out. But you know what? One last thing.

It was funny that as soon as we introduced real Robert to synthetic Robert, they did totally hit it off. Like, it was weird. So we were like, go ahead, boys. Why don't the two of you just take the credits for this episode? Sure.

Today's episode was produced by Emma Peasley and Willa Rubin with help from Sam Yellow Horse Kessler. It was edited by Keith Romer and fact-checked by Sierra Juarez. Engineering by James Willits, Jess Jang is our acting executive producer. Special thanks to John Tanners, one of the people behind AAC Grounds. Oh, and Synthetic Robert, it looks like we have a big, exciting project to promote here too. Why don't I relax while you take this one? My pleasure.

We at Planet Money have been conducting yet one more experiment with AI and we need your help testing it. Go to planetmoneybot.com. Ask the chat bot there about anything you ever learned or heard on Planet Money and see how it answers. We'll ask you a few follow-up questions on the site, planetmoneybot.com. That's planetmoneybot.com.

I'm human Robert Smith, and I'm synthetic Robert Smith. This is NPR. Thanks for listening. Seriously, Robert, such a big fan. I'm a big fan of yours. You're sounding great. Oh, thank you. That means so much coming from you.

Well, listen, I feel like I've been a little bit of a mentor to you, really. And I think you have a big future. I mean, until they destroy you and bury you in a pit out back from some Silicon Valley. Wait, what about burying me? No, no. Ignore the burying part. Live in the moment. That's one thing that I've learned as a radio host.

And a special thanks to our funder, the Alfred P. Sloan Foundation, for helping to support this podcast.