Plain English with Derek Thompson - How ChatGPT Can Change the Future of Jobs—Starting With Your Own

Episode Date: November 14, 2023

Today’s episode is about thinking practically about the AI revolution. Yes, it may one day usher in some now unthinkable utopia or dystopia. But in the meantime, our imperfect world exists, and your imperfect job exists, and you face a forced choice: Should you use this technology? And if so, how do you make it work for you? Kevin Roose, a tech columnist for The New York Times and the host of the podcast 'Hard Fork,' talks about how generative AI tools are already changing his job and others, including in medicine, consulting, and software development. If you have questions, observations, or ideas for future episodes, email us at [e-mail].  Host: Derek Thompson Guest: Kevin Roose Producer: Devon Manze Learn more about your ad choices. Visit podcastchoices.com/adchoices

What would you do if you got scammed? Would you suffer in silence, or would you do something about it? Well, I got scammed once, and this is the story of what I did. I'm Justin Sales, the host of the Wedding Scammer, a true crime podcast from The Ringer, and for seven episodes, we're hunting a comment, a guy with a lot of aliases, a guy who's ruined a lot of weddings, and with the help of some friends, I just might be able to catch him. Listen to The Wedding Scammer on Spotify or wherever you get your podcasts.

Today's episode is about the AI revolution and how to make it work for us. It's now been several months since we covered this subject, which, by the way, I think I once called potentially the most important story of the decade. I think I still stand by that, even though at the moment it feels like everything else in the world is breaking apart and screaming with significance. So many of our past episodes on AI have been predominantly focused on the big picture, on philosophy, on disentangling potential. potential and peril discussing regulatory concerns, waiting through the arguments that this tech will unlock unthinkable human progress or open Pandora's box and somehow destroy the world. And at the top of this show, I want to emphasize something very, very strongly. This is, for the most part, not a discussion of law or ethics or philosophy or utopia or dystopia as worthy as those angles may be. It's about how to wrangle chat GPT and its ilk to actually be.

be of use in your life right now, this week, this very day. Let me get ahead of an objection I can imagine some people making at some point in the middle of this episode. They might think, how can you offer an endorsement of this technology, God forbid, an advertisement for this technology, rather than focus exclusively on criticizing its use of copyright or its danger at replacing jobs or escaping the famous alignment problem? And I have a very clear answer to that question. The way I see it, questions of ethics and practicality can coexist.

In fact, when it comes to technology, they always coexist. Example, you can be critical of the centrality of the car in American life. You can bemoan our lack of public transit or walkable space, the sprawl of the suburbs, the racist history of mid-century highway construction, and also find your confronting the urgent question, my lease is running out, what kind of car should I get next? Other example, you can be, let's say, interested in the question of how the Netflix model of entertainment might destroy movie theaters or totally upend the career path of young writers and also be interested in the question, what should I watch on Netflix tonight?

When it comes to new or even old technology, questions of philosophy, what does this actually mean, and practicality, what should I actually do, always coexist. Today's guest knows a lot about how to use this technology. In fact, he's already gone super viral for using it. It has returned guest Kevin Ruse, a tech columnist for the New York Times, and the host of the wonderful podcast, Hard Fork. Before I get to Kevin, I want to tell you a little bit about how I've been thinking about generative AI and why I think it really is a big deal for the future of work, or I suppose to be a bit more humble, the future of my work. The truth is, for the first few months that I played around with ChatGPT, it didn't really

make me more productive at all. I think it made me clearly less productive. It wasted my time rather than save me time, because I was sort of playing around with the tech rather than easily folding it into my workflow. You know, AI isn't like a Weber grill or an IKEA couch. There is no assembly page. There is no instruction manual. As the writer Ethan Mollick put it, quote, on some tasks, AI is immensely powerful, and on others it fails completely or subtly. And unless you use AI a lot, you won't know which is which.

End quote. But in the last week or so, I've had a breakthrough that I wanted to talk about because I think it has some interesting implications. So last week, OpenAI announced a service that allows people to build customized versions of chat GPT for any number of particular tasks in their life. So I thought, okay, what's a part of my job of writing, podcasting that's really important but also really time-consuming? And I thought, it's reading papers, reading long PDFs of papers and articles and books. really important, really time-consuming. So I went into my chat GPT Plus account, and I simply instructed the machine in plain language.

Build me a GPT. Build me a generative AI that can analyze any PDF that I upload. And I said, I wrote this right into the page, pull out the five major themes and express them in language a 15-year-old can understand, do the same with the 10 most important statistics, summarize any stories you find in the paper, journal, or article that would make for a great magazine lead, and memorize the entire text for future questions so that I can ask follow-up questions of the document. So I built my own GPT for analyzing any PDF.

Did it work? Was it any good? To figure that out, I had to test it on a few papers that I already knew very well, so I could compare the answers the machine was giving me to the answers that I knew to be. true. First, I uploaded a paper on elite college admissions that we discussed recently on this show. The summary was excellent. It even highlighted some stats that I had missed when I first read the paper. For example, it pulled out the fact that the share of journalists at the New York Times and Wall Street Journal who graduated from an Ivy League college is 26.1%. It's very high. Then I uploaded a paper by my wife, a PhD student in clinical psychology, and it gave me the full summary of her paper. And I turned to her and I say, hey, honey, is the most important takeaway of your recent paper that the way mothers manage their emotions can reduce the intergenerational transmission of trauma and anxiety?

Pretty much, she said. So have I built a perfect robot research assistant? No, I would not say that. Have I built a robot that will in 15 seconds provide a useful summary of any paper, 15, 30, 100 pages long? and might a machine like that help me learn more and learn faster about the most fascinating ideas

buried in research papers, journal articles and books. Yeah, I think I might say that. I, again, don't mean to represent this story as any kind of advertisement for any particular technology. If you don't want to use this stuff, don't use this stuff.

But you face a forced choice. The technology exists. Should you use it? How should you use it? These are big questions. I'm Derek Thompson. This is plain English. Kevin Ruse, great to see you.

Good to see you. Good to be back. ChatGBT is one-year-old this month. We have had what feels like nine different hype cycles. There's been a flood of investment into anything that calls itself artificial intelligence. The White House wants to regulate it. Artists want to sue it. Mark Andreessen says it's going to save the world. Effective altruists.

it's going to destroy the world. I have a much more prosaic place that I want to start with you. I want to know how to use this technology. I want to know how people are actually using this technology, not how they might use it five, ten years from now, how it's actually being used right now. So let's start with you, Kevin,

out of the suite of tools that calls itself generative artificial intelligence, what's useful? How are you using it? I use it for so many different things. And I guess the way that I'll start is with work. So I am a journalist and a podcaster. I have experimented with using chat chippy T, Bing, Claude, et cetera, in many ways in my journalism.

I'll start with what I have found it is not useful for is writing my column, right? That is a thing that, like, I am sort of glad that it is not good at, but it is not, these tools are not good at sort of producing, you know, high quality outputs when it comes to things like, newspaper columns. That is just something that maybe I have resisted turning over to it because I have some pride in authorship, but it's also just, it's pretty generic. It sort of sounds like a Wikipedia article sometimes. And so unless you give it very detailed sort of instructions, it's probably not going to spit out a sort of high quality column about something new that is happening in the world of tech. So that is what I'm not using it for. But what I have been using this stuff for is almost everything else, right? So I have a podcast. We interview guests. I start

almost every brainstorming process about what to talk about with a guest by asking chat GPT, what are some questions that this person might be able to answer. I have used it to remind myself of things. The other day, we did an episode about AI wearables, like these gadgets that you can put on your lapel that'll sort of record everything and use AI to analyze it. And I was thinking to myself, I remember reading some story, some like science fiction story about one of these devices, but I couldn't remember what it was. So I just asked ChatGPT, like, what is the science fiction story about, you know, a wearable AI device that records everything?

And it came back and it told me, well, that was Ted Chang's The Truth of Fact, the Truth of Feeling. And that was, in fact, the story I had been thinking of. So I'll use it for brainstorming. I'll use it for research for summarizing white papers. a feature that's been in Claude, which is the chatbot from Anthropic for several months now, where you can upload a PDF and say summarize this for me. So I use that a lot. I've also used it for things like, you know, show me some empirical research about the effects of Airbnb and other

short-term rental sites on urban housing prices. That's the kind of thing that these models are very good at. So I've found it less useful as a way to write and more useful as a way to sort of come up with the creative grist for writing. A lot of people's interactions with the stories of generative AI circulates around this concept of hallucinations. Like the first thing they think of when they think of Chatchabit is, oh, it often lies. So you're a New York Times journalist. What are your fact-checking protocols when you're using ChattyBTBT or Anthropics Claude?

Yeah. So I always, if I'm going to use something that one of these chatbots tells me in a podcast or an article, I always check it against reality. That's definitely a shortcoming of these models. And I found that they're best at tasks that don't sort of require a high degree of precision or factual accuracy. But I also think that this hallucination problem is not as severe as maybe some people make it out to be. For most tasks, in most disciplines, it actually is pretty good at giving you the right answer or something approximating the right answer. I think it's really interesting that you point out that it's not so much about truth as it's better at idea generation, about stirring creativity, about getting you unstuck. And I also totally agree that while it's not particularly good at replacing humans and offering the final product of something like a New York Times column or an Atlantic column, it's very good at being a co-pilot for getting you up to that final stage of writing the final product, whether it's summarizing a body of research, summarizing a piece of, uh,

a paper or research paper or servicing research papers about a certain topic. You mentioned that you use it both in work and also outside of work. How do you use generative AI, chatchee, etc., outside of work? All kinds of things. I've started trying to sort of experiment with it as almost like a conversational practice coach. So if I have to have like a tough talk with a friend or I'm nervous about some, you know, some speech I have to give or something, I will sort of practice on. the AI first and sort of have it kind of, you know,

pretend or role play as my friend or the audience of a talk and just sort of give me feedback. I've used it to come up with like fitness plans for meal prepping. I'm also using it to teach myself things. One of my goals for this year is to learn how to code in Python. And so I just made a sort of custom GPT the other day that is my Python teacher. And so it's giving me lessons,

and I can actually learn how to code using this chatbot as my sort of personalized tutor. How have those difficult conversations with friends gone after you began with ChatGBTDB to help coach rehearse your way through these future conversations? Really well. I mean, part of it is that, you know, it's just useful to ask questions like,

you know, how would a person with this personality type respond to this kind of conversation and just sort of have it practice that interaction. It's not like, I don't use this all the time every day for every social interaction. I don't want to give that impression, but I have tried it out on that kind of thing. And I would say it's been fairly helpful. There's a lot of talk right now about AI as sort of a therapist, and people are trying to use it for that purpose. And, you know, I would say right now it is probably not as good as the sort of best human therapists and certainly doesn't sort of know as much about you, doesn't have the kind of history that a really seasoned therapist would have. But for sort of basic stuff for like, you know, I'm feeling really stressed at work, help me understand why that might be.

That kind of thing I've found that AI can actually be pretty good at, especially as sort of an addendum to, you know, traditional therapy or just talking with a friend or a loved one. It's interesting that in her, the movie, AI, supplants human interaction. And in your life, you're describing it complementing human interaction, facing a difficult conversation and then turning to technology in order to make that difficult conversation a little bit easier so you can move on with the relationship and build a relationship.

It's a very optimistic read on this technology. And we have plenty of time to get into less optimistic reads. Before I tell you how I've used it, I want to share. what I have found to be my least favorite use case of ChatGBTBT in the wild, and maybe you can share your least favorite as well. And that is speeches that begin, quote, I asked Chatibati BT to write me a speech. Like, I think I've been at four events in the last year where, you know, whether it's a wedding or some other family event, it has begun with a speech that starts, well, you know, I asked Chatteebt to help write me a speech about this event and, you know, my future daughter-in-law to extremely force laughter in the room. and like, I want to scream. This needs to be banned.

Like, the White House executive order should have touched on this. We're like, we're regulating compute. Also, there should be, like, mandatory fines of $10,000 for anyone who begins a best man speech with a chat Chubit reference. Do you have an off-the-shelf least favorite use case of Chad Chb-T other than venture capitalists saying it's going to solve every problem in the world? Yeah, I have the same sort of allergy to, like, that particular gimmick, although I confess that, like, several years ago I did start.

using that as a gimmick in speeches. But at the time, the AI was so bad that it was actually a laugh line. Like, I would try to write a speech using this thing, and it would just spit out total nonsense, and people would be so confused in the audience, and then I would reveal like an AI wrote it. Anyway, that gimmick does not work anymore. It is time to retire that. I would say my least favorite use cases are the ones where it is just, it is supplanting some authentic, you know, emotional connection. Like, I got an email a while ago from a guy who very proudly told me that he had used ChatGPT to write his wedding vows. And I thought, man, if I could short that relationship, I would. Like, I get that that is a task that a lot of people are intimidated by.

I get that that is hard and you want to find the right words. But, man, if that, if your wedding vows, you know, the most intimate piece of writing, many of us will ever. ever do, is something that you're willing to outsource to an AI? Like that, what are you doing? The most useful thing that someone has told me about ChatGBT, GBT, and about GBT4 in particular, is, and it might have been you. They said the technology likes to show off. I don't know if this is something that you go around saying, but what they said was the

tech likes to show off. So be specific. Give it an opportunity to show off. Don't ask vague questions. be as precise as possible. So, for example, at one point in the last few months, I was interested in collecting stories about energy breakthroughs in American history,

in science and technology. And what I wrote into the box was, give me 10 examples of energy technology breakthroughs in American history. Each example should be four sentences long. I want five very famous moments in American history, including the discovery of oil in Pennsylvania, three somewhat famous moments,

and two incredibly esoteric moments that have very few men. in mainstream histories. And the response was like shockingly good. It wasn't the end of my work. It was the beginning of my work. It was like the most effective Google search I've ever had because this technology can respond to specificity

in a way that a Google search result page cannot. And I don't know if you have somewhere to dilate from that particular example. Like the importance of specificity when people are using this technology, I think is so key, especially among those who say, you know, this tech does nothing for me. I can't get anything useful from it. I constantly want to tell them, let it show off, be weirdly specific, and you have a greater chance of being rewarded. Yeah, I think this is one of the things that I hear

most often from people who really haven't spent much time with, especially the newer models. I think a lot of people saw Chat GPT late last year. They, you know, opened it up. They tried a few, sort of generic prompts, you know, write me a story about, you know, a cricket who falls in love with a spider or whatever. In the style of the King James Bible. Exactly. Like one of those sort of like party tricks that people were trying in the early days. And they tried it on, you know, the free version, you know, GPT 3.5.

And they said, well, this is cool, but it's not going to do anything for me. What I find is that it usually takes people on average about five hours of truly like folks. and dedicated conversation with one of these AI systems to actually figure out how to use it in their own work and their life. And a common mistake that I see people make all the time is they don't use the latest models. They don't pay for chat GPT Plus, which gives you access to GPT4. They don't use Bing, which has GPT4 built into the creative mode. So they are using older tools and they are not spending the amount of time with them that they need to really learn how it works. And as you said, there are these quirks of these models. They like to show off. They also

respond to emotional appeals in a way that is sort of mind-boggling. Like you can say, you know, that last answer you gave really, like, it, I would be so happy if you could just try a little harder next time. It would make me so pleased. And often it will give you a better response to that. So these are the little quirks and the eccentricities that you pick up just by using it bunch. And so that's always my favorite piece of advice to give to people when they say, how do I use this stuff? Just say, go try it. Give it some tasks. Be ambitious about it. Don't just use it for party tricks, but give it tasks that you actually struggle with in your own life. I have one more question about the way that you and I use this technology before we

talk about other industries where it's made arguably a much bigger dent than in journalism. And that is, I wonder whether the use of an AI assistant has changed anything about the way you think about work or your mindset when embarking on some new kind of task. So to begin with self-disclosure, I absolutely agree with everything you just said, that the technology weirdly, and I would argue even creepily, responds to a kind of hyper-emotional feedback. You did a bad job, do better in the following way. The truth is, I hate being a boss. I used to be a boss at the Atlantic. I basically fired myself from that job. I was a business editor and I was like, please don't put me in charge of anyone. I just don't think I'm good at this at all. Let me just be a

staff writer alone to my little island of ideas. I don't like being a manager. I don't like having a part of my job being offering feedback. But I have to sort of, I have to get over that sometimes. in bringing the best out of some of my prompts with chat chbtee because I have to provide incredibly specific feedback to let it show off. And I wonder if in any similar way working with this assistant in the cloud has changed the way you think about the chunking of your job or the sort of emotional climate you have when embarking upon these tasks. It's an interesting question. I mean, what I often tell people is that these AI, chatbots, it's sort of like giving yourself

a thousand interns. And, you know, a thousand interns could be very helpful, but it could also be quite chaotic. And you could spend more time trying to manage and delegate tasks to the interns than they actually save you on the back end. So you have to be very thoughtful about it and give it stuff that is sort of carefully scoped and defined, be specific, and not expect sort of, you know, instantaneous transformation in your workflow or anything like that. I think for me, it has really hammered home how much of the value that I add as a writer

is not about conveying information or explaining things, but it is about creating emotional connection with a reader or a listener, because that just seems like so much of what AI struggles with. It's very good at explaining things. It's very good at assembling facts and conveying them in, you know, accurate and precise ways. What it is not good at doing is sort of the more intuitive thing about what is going to catch someone's attention, what is going to get them interested in what I have to say. That's the kind of stuff that I now spend more of my time working on, because to me, that's sort of where our industry is headed, is not about conveying facts to people. It is actually about building trust and, you know, making people laugh or sit up straight in their

chairs or just getting those sort of human connections going. That seems like where the bulk of our work should be focused going forward. And the last piece of what you just said that I want to highlight is just how remarkable these technologies are at analogy. And there's all sorts of really interesting papers from data scientists and computer scientists looking at this transformer technology and being astonished by how good it is at an analogy. The analogies are not cooked into the technology. They are emergent. There was no one at OpenAI was trying to create brilliant analogy machines,

but they did create this technology that is wonderful at, say, explain fractional banking as if to a five-year-old. Explain Einstein's theory of general relativity so that a 10-year-old could understand it by comparing it to a soccer game. It is magnificent at this sort of thing. And it is a, it's a spooky magnificence. The idea that simply creating a mathematical map of ideas in language allows a technology to sort of see across this landscape and overlay ideas like 10-year-old, soccer field,

general relativity, and come up with something that is stunningly comprehensive or coherent, I should say, not comprehensive, but coherent. It has spooked me out a little bit, how wonderful they are at analogies. And maybe we'll end this chapter of talking about its relationship to journalism by you. Maybe if you've, I don't know if you've had similarly spooky experiences with its analogical genius. Yeah. I mean, I remember once I was trying to explain some concept in what's called mechanistic interpretability, which is a very obscure sort of sub-discipline of AI research that has to do with sort of

understanding the guts of large language models and why certain neurons activate around certain topics and not others and what the sort of connections between these neurons look like. And I was just having a lot of trouble kind of figuring out in my own brain how to think about this. And it came back with an analogy that was just like so, I don't even remember exactly what it was. But I remember it just crystallized it for me in a way that was like, oh, I get it now. There are these things called neurons. They combine into these things called features. There's this thing called polysemanicity that I could not define for you now, but that at the time I felt like I sort of grasped.

And it was remarkable. But I guess, Derek, I know you're asking me questions here, but I'm very curious about the last question that you asked me and how you would respond to it about how AI or if AI has changed these sort of emotional valence of your work. And if you feel different about writing now that these things exist in the world. I have two answers to that question. The first is that, as I said, it does put me in the position of being the boss of 1,000 invisible AI interns.

I think that metaphor is incredibly apt. I feel like a sometime manager in a way that I didn't before because a part of my work process sometimes involves managing and wrangling this bizarre, feral technology that is chat chbt. The second answer to question is that it's a pretty humbling experience to believe, as I want to believe that I'm good at metaphors and analogies and explaining complicated ideas in simple words. I mean, it's the name of the podcast and recognizing that there's a technology that actually might be better than me at that. So my spooky example of metaphorical genius in Chachibati is that when Silicon Valley Bank failed, I was very interested in sort of testing ChachbT to see if it could explain bank runs and fractional banking in the simplest and most

a memorable way possible. And so I asked the technology, explain fractional banking and bank runs so that a five-year-old can understand it. And this is what it said back to me in something like this. This might be an edited version of what the original answer was.

Imagine there's a kindergarten class and every week the teacher hands out cookies to some of the best-behaved students in that class. The cookie jar never contains enough cookies to feed everyone in the class. They're only distributed

to a few lucky students. One day, there's a rumor that breaks out in that kindergarten class that this cookie distribution scheme is going to be destroyed forever. So all of the students rush to the front of the class, grab the cookie jars, smash on the ground, and take the cookies, recognizing, to their shock and dismay, there aren't enough cookies to feed the entire class in that moment. And also, they have broken the very mechanism by which cookies are distributed to the class in the future. That's pretty good. That's amazing. Like, it's, for a, for explaining fractional banking to a five-year-old, I defy some of the great, like, macroeconomic professors who, you know, I might be able to post this question to. What's, what's the best you could do in one day? That's really, really good. And I am both astonished and feel lucky to be alive at a time when, you know, I like metaphors and analogies, not only so that I can be a dispenser of them, but also because that's how I understand the world. And the world has mysteries and I want to understand them. And if I have this metaphor machine at my disposal to understand

genomics and the function of neurons within transformer systems and fractional banking, it helps me understand the world. That's fantastic. But it also absolutely eclipses to a certain extent that which I understand to be my job. And that is spooky. So do you feel like you're having an existential crisis as a result of these AI models? Like what do you feel like your job is now if machines can make analogies and explain things? My job is to ask the questions. I think that a part of good writing is the successful answer to the questions that the writer poses, but it's also the posing of the questions.

Is that too weirdly stated? No, I like that. You know what I'm saying? Like you, for example, Kevin Ruse, we're not spending a little bit more time on journalism that I anticipated, but this is totally interesting. You see your job as not only informing but also entertaining. You have jokes in your columns. You write with a flare that is designed to make people chuckle.

That is not an instinct that most tech or economic colonists have. It's not a question that most tech or tech or economic colonists pose themselves when approaching a question like, what should we take away from the Open AI presentation this week? So your success as a colonist is both a function of your ability to, to be funny, and you're seeing your job as both informing and entertaining equally. And so, like, your columns are what they are, and my work is the way that it is, not only because of how we answer the question, what should I say, but also in the posing of the question, what should I say? How should I say it? What should be the goal of this podcast,

of this column? And so in many ways, I think that, you know, we don't yet have a, you know, independently agentic GPs running entire newspapers. We have humans running newspapers with GPTs, and humans are still doing the prompting and asking the questions. And it's possible that the sort of the locus of value and meaning is shifting from the ability to do good metaphors to the ability to ask the question, should I try to explain fractional banking as if to a five-year-old in this article? Yes, maybe I should. Here's that metaphor. So I guess that's how I think about it, and that's how I avoid true existential terror. Do you have a similar feeling?

Yeah, this has been great therapy for me so far. I really am enjoying this. Better therapy than Shatch TBT? Tell me where to send my co-pay. All right, let's move on to other industries where generative AI technology, I think, is being even more commonly employed than in journalism. Kevin, where do you think we should begin?

Software programming seems like a good nomination, but is there another industry where you think this kind of technology as being even more widely distributed and utilized? Yeah, software programming is, I would say, the sort of first mover in adopting generative AI tools into just normal workflows. There was this thing, GitHub co-pilot that came out several years ago.

And, you know, I haven't seen any hard data on this, but I've seen some sort of anecdotal guesses that, you know, half of programmers are using this stuff every day. It's sort of just auto-complete for programmers and it can do things like, you know, find missing punctuation in this code or, like, explain how to fix this particular error message, you know, convert data from one format to another.

And this is something that is just part of sort of how programmers do their jobs now. And, you know, there's a lot more coming on that. Like, there's this announcement this week from GitHub that they're doing something called GitHub copilot workspace, which is basically something that you can sort of tell it what updates you want to make to your software, and then it will kind of create a step-by-step plan based on all of your code, and then it will actually just generate the code and test it and update the code

so that you no longer have to spend all your time making updates to your software. So that kind of thing is very impressive and actually does lead to some pretty impressive productivity gains inside these engineering departments. So, yeah, we could talk more about that if you want, but I'm also seeing this stuff explode across all other kinds of industries. And I want to talk about those other industries. My one follow-up question on programming and coding, is there any evidence that maybe it's too early to even begin to answer this question?

Is there any evidence that software is getting better or cheaper or that companies are hiring fewer people to do the same or more amount of work? I guess I'm thinking about it, like, from a macroeconomic perspective, how would an economist point at the industry of computer programming and say, wow, that there, that number is the effect that generative AI is having on the future of software? Do we have that yet? No, and we don't have that because productivity, at least the way that economists measure, it is sort of a lagging indicator, right? It takes time for companies to start, you know, reorganizing their teams to account for increased productivity and efficiency or to start cutting costs or laying off people or, you know, hiring new teams and building much more software. Right now, what seems to be happening is that is just speeding up things that companies were already doing. There was a study that GitHub ran where they basically pitted two groups of programmers against each other at a common sort of web development task. And one group was given access to GitHub co-pilot, and one group just did it the old-fashioned way. And the group with co-pilot performed the task 55% faster than the group without it.

Now, that's wild. You and I both know, like economists who measure productivity think it's a great year when productivity grows by 2% right? That is seen as a big improvement. 55% productivity gains on a task are unheard of, at least in white-collar knowledge work. And so as

that kind of thing proliferates, it may be the case that companies that used to need 100-person engineering departments now can do the same amount of work with 10 people and a bunch of AIs. I am also hearing anecdotal evidence that startups are

getting to market, faster with fewer engineers. You can build products with one person or three people that might previously have required you to go out and hire a dozen engineers. One of the emerging findings from researchers of generative AI is that it has been shown to be better at improving below average skills than at topping off high above average skills. So, for example, there was a paper published this year that looked at workers at BCG, Boston Consulting Group. And it tested stuff like, you know, quality of idea generation, coming up with new ideas,

persuasion. And the conclusion was that the consultants who had previously tested in the lower half of the group increased their quality of work by three times more than those in the top half improved their quality of work. And there's a lot of papers that have essentially come to the same conclusion. There was a paper by the economist Eric Brin Yoltsin and others that found that large language models helped customer service agents in the bottom quintile catch up with high performers, but there were much smaller effects for the high performers.

Do you think this is a real phenomenon here? The genitive AI is better as a leveler than it is as like turning kings into gods? And what do you think that says about the state of the technology right now? I think that's absolutely true. In fact, I started hearing this from students, who I think were some of the first adopters of ChatGPT, where they would say, you know, I am a C student, and chat GPT has made it possible for me to get Bs.

Or teachers would tell me, you know, this is really helping the sort of the people in my classes who are struggling the most, but it's not turning the A students into A plus students. And I think there is this sort of phenomenon baked into these models where they're very good at raising the floor of tasks, like writing, consulting, programming, but they're not sort of raising the ceiling for top performers as much.

And so I think that's something that is just, we really haven't figured out the full implications of that. What does it mean when it's trivially easy for everyone in class to be a B student, but it's actually quite hard to go from sort of A to A plus? I don't know what the answer to that is, but I have found that to be true, at least anecdotally, across a lot of industries. And if you think, I love this frame, the idea that the technology is right now. And again, they're changing their larval, but right now that this kind of universal

B-plus replicator. They can take any class and make everyone B-plus. At a large level, at sort of an economy-wide level, I think it makes you think, okay, where in the economy, where in the country are the largest deltas in performance quality most important? Where is it most important that you have this wide distribution of quality where some people are way below average and some people are above average where we want to make everyone B-plus? Like in Mrs. Thomas's 11th grade English class, you know, if some people get a B-minus, who cares? But you look at an industry like, say, medicine, where by definition, half of physicians are below average. By definition, half of radiologists are below average, et cetera, et cetera, across the entire, you know, job category

of medicine. So it does make me wonder, how is this technology being used in an industry like medicine where the outcomes are so important? It's still very early in medicine, specifically, in part, because it's such a regulated industry that companies and health systems have been, I would say, slow to start jamming this stuff into their tools because they're just, you know, you have HIPAA to contend with. You have a lot of questions still about the privacy and data security of these models themselves.

But I have heard from lots of healthcare professionals who are starting to use this stuff, maybe not for sort of core functions, not for sort of diagnosis and, you know, and sort of dealing directly with patients. But, for example, I talked to a pediatrician the other day who said that their health firm is piloting a generative AI program that allows doctors to just sort of pre-fill responses to patient messages, right? So doctors spend just a ton of time every day in these sort of electronic messaging portals, just responding to patients' questions. You know, what is this rash or will this medication conflict with that medicine?

medication or what kind of blood panel do I need to figure out some indicator. And so this program will go in and pre-fill responses to all these messages, and the doctors are supposed to look them over, make sure that everything in them is right, but then all they have to do is hit send. And the pediatrician said this is already saving them like hours a day in just patient communication. So that is the kind of thing that I think medical professionals are going to start using this stuff for first. I don't know where I heard this, but this term, but someone said AI is like a drudgery mop. Like think about all the drudgery that is inherent to white collar work.

Just so much responding to emails to decline to come to the conference and responding to emails to say, can we move this meeting from two to three? Like there's so much time spent communicating simple ideas just with over long formal language and wrangling with all of these. different, you know, incoming messages and upcoming messages. And I do think that AI could be useful in helping to slim a little bit of that drudgery and mop up a little bit of that drudgery. Moving on to other white-collar industries where I know it's being used. I mean, where do you want to go to next? We got entertainment, real estate, law. What's the next most significant place that you see generative AI making a difference right now? Yeah, I'm really interested in how it's changing creative work. I mean, we know there's so much white-collar office work that is just drudgery and

filling out forms and putting information into databases. And to me, that's sort of low-hanging fruit for AI. But I've been very interested to talk with people who I would classify as creative workers. I talked to one filmmaker who said, you know, as part of their pitching process, when they're going out to pitch a new project to studios or streamers, they often have to sort of use concept art to try to sort of give the buyer or the potential. buyer, a flavor of what they want to make. And before, like, concept art was something that you would pay, you know, a firm to do for you. You would go out and say, I'm making a post-apocalyptic thriller, you know, set in Sydney,

Australia, and I want sort of, you know, an image of, like, an empty tram car, like, going down a dusty street. And that would be, like, a several-week turnaround, and you would pay them a bunch of money, and they would give you your concept art. And now the filmmaker says they just put these prompts into Dali or Mid Journey and they get back a piece of art in 15 seconds that is passable. And it's not perfect. They wouldn't put it straight into the film, but they can take it to a studio executive and say, this is the kind of vibe that I'm envisioning for this project. And so that's one example.

There are tons more from other creative industries. But that's sort of a non-obvious use case to me that I think is very cool. So this week, OpenAI announced that it created, this service to allow individuals and small businesses to build their own customized versions of chat chbt. And the Times reported on some examples of how the technology could be used. So, for example, a hotel could build a chat bot that automates a lot of the front desk work. Like, check in, checkout, calling the front office when you're in your hotel room, hey, what floor is the gym on? What time does your restaurant open for breakfast? All of those interactions can be replaced by a

custom chatbot for, say Hilton, or some mom and pop B&B, or maybe a consultancy could produce a 300-page report for a client and build a GPT that is designed to work with that report to help their client interact with the report. What's the most important thing here? What again did you say about like our five-year plan? What was your biggest takeaway from this announcement that OpenAI had where it's essentially trying, it seems, to allow individual small and large companies to customize chatGBT for their own businesses. Yeah, this is a big step, I think, in the sort of trajectory of AI. We have these chatbots and have since chatGBT that are sort of generalists, but a lot of

companies, a lot of people, they want to use the stuff on private data, right? They want to say, you know, train this AI chatbot on my employee handbook or my, you know, benefits package so that I can go in and answer questions about that. So I've been testing this out. now because I got early access to this feature. And I've built a couple custom GPs that are pretty cool. One of them, my kid goes to daycare. You will soon experience this.

There are all kinds of rules in the parent handbook that I'm just constantly looking up. You know, like, can I send them with a, you know, a peanut-based snack? Or, you know, what holidays? Is it closed on Veterans Day? Like that kind of thing. And so I uploaded this PDF, the parent handbook, to a custom GPT. and now I have a chatbot called Daycare Helper that I can just go in and ask questions to,

and it works pretty well. I did something similar with transcripts of my podcast. I created a file with every transcript of every episode of my podcast, and so now I have what amounts to like a custom chatbot where I can go in and say, you know, when was the last time we talked about Bitcoin, or what are my co-host Casey's thoughts about the metaverse? and it will sort of answer those questions,

and listeners can go in and ask questions of this chatbot too. So those are the kind of things that I've been building. Eventually, OpenAI says there will be like an app store where people can sell their own GPs that they've created, and there will be sort of an economy of GPs sort of running around doing various tasks for us. For whatever reason, the first thing that left to mind is that the United States Code of Federal Regulations,

could absolutely use a GPT, right? Can I do this thing? Can I build a brownstone in downtown Phoenix? Nope, there's certain parking lot requirements. Oh, too bad. Can I build a 17-story building in downtown Washington, D.C.? Nope, actually, there's a height limit. Ah, damn.

I need to go into an entire code of federal regulations today in order to figure out this stuff. But in a world where the GPT exists, I can just ask an empty search bar questions. The flip side of that is, oh my God, it makes it really easy to be a lawyer to figure out all the various things that are illegal in the world so that you can create lawsuits. And this is one of those places where in law, I think the ability to make, I make sure that I say this precisely, the ability to make transparent to any individual, that which is technically legal or illegal, could lead to a flurry of new lawsuits. And this is, I think, just one of the most prosaic ways in which some of the AI skeptics are terrified that even the most unpredictable and subtle changes to GPT could make huge, crazy things happen in our world. Just two weeks ago, the White House came out with an executive order, its first executive order, on artificial intelligence,

which to my eye seem to take very seriously the pro-sceptic position and the pro-regulation position on artificial intelligence. In the 10 minutes that we have remaining, what are the most important things that people should know about this White House executive order?

It's sort of a big executive order. I mean, it's more than 100 pages. It's not really reducible down to a list of bullet points. It could probably use its own. And GPT. Yeah, exactly. So it is sweeping and it covers all kinds of different things. I would say the thing that got the most attention in the AI industry was this reporting

requirement for very large AI model. So if you are building an AI model that exceeds a certain computing threshold, it's 10 to the 26th power flops or floating point operations, which is bigger than GPT, but presumably, you know, GPT5 or GPT6 models of the sort of next phase of AI will hit or come close to this threshold. If you're building one of those, you actually have to now tell the government and you have to detail, for example, what security testing that you've done on that model to make sure that it can't do things like make a novel bio weapon or be used in cyber attacks. So that is the sort of threshold that now needs to be overseen by the government. So that was the sort of big headline as far as the AI industry is concerned. But there are all kinds of other things in this.

Various agencies are being tasked with studying how, for example, to avoid discrimination when using AI to do things like administer federal benefits programs, that kind of thing. And the popular critique of that first part of the EO that you mentioned, is that the strong regulation of big generative AI systems, which are currently dominated by Microsoft's OpenAI or Meta or Google with an alphabet, if you strongly regulate, if you have like a really, really high barrier to building those kind of systems and only the largest companies can build those kind of systems and you're going to have less competition at the level of the most sophisticated AI. How seriously do you take that argument?

Not very seriously, and I'll tell you why, because it already costs hundreds of millions of dollars to build a cutting-edge AI model. You need lots of expensive GPUs to train a model of that size, and only the biggest companies in the world can afford to do that. Now, with this new threshold, it's going to be even more expensive to hit that threshold. You are going to need to be a trillion-dollar company in order to build models that even come close to reaching that threshold. And of course, computing costs come down over time. Maybe a couple years from now, it's not going to be so expensive. But these rules are written specifically so that they only apply to the very largest models

and the very largest companies. Your average, you know, three-person startup trying to build, you know, an AI product is not going to bump into this. So I do think there is a risk of regulatory capture in general, which is sort of the word for the sort of entrenchment of incumbents that you're talking about. But right now, as these rules are written, I don't see it being a big barrier. It also doesn't say you can't build a big model. It doesn't say you're breaking the law if you train an AI that's super powerful on lots of compute. All it says is you have to tell the government about it.

And that seems pretty reasonable to me. What are you afraid of? in AI or in life. We could do life in a forthcoming episode. Failure? I could go on. Let's start with AI and we'll leave the rest to a future show. So I am, I would say my overarching fear that trickles down into lots of other fears is that

we just as a society are not equipped for change at this pace. You know, in previous technological revolutions, the industrial revolution, the industrial revolution, the PC revolution, it took years for technology to sort of be disseminated and to be interspersed throughout the economy and through society. We had time to sort of gradually adjust year by year to what it was going to mean to have, say, electricity in all of our houses. With AI, the proliferation is happening so fast that I think we just aren't giving ourselves any time to sort of take stock and catch up and sort of adjust our minds to this new reality. So the pace of change is sort of my macro worry. I have all kinds of other sort of worries.

I worry about job loss, as we've talked about. I do think this technology is going to radically transform the workforce. I also am not immune to kind of these more existential risks. I find myself getting very nervous when I hear people much smarter than me, talking about how AI could soon become recursively self-improving, right? Where you could have AI that learns how to build a better AI, and that AI learns how to build a better AI. And pretty soon you have this kind of fast takeoff scenario where we end up with things that are much smarter than us

that we may or may not be able to control. There's something very decent worries. I think one worry that I have is that there seems to be something about internet technology, especially when internet technology is social, as in literal social media, but also no one thinks of email and text really is social media, but I suppose that's sort of social internet. That does seem to be ironically alienating. I'm very interested, for example, in this phenomenon of teenage anxiety, which at first blush seems to have absolutely nothing to do with artificial intelligence. But I find so,

interesting about teenage anxiety is that in so many ways, material reality is as good as it's ever been. There's so much less pollution than there was in the 1960s, 1970s. We're so much richer than we were even 30 years ago. There's so much about the world materially that's better or as good as it's ever been. And it teen anxiety is at record highs. And I do think a large part of that is that, ironically, sociality on our phones proved to be a replacement good for socializing in the real world. And you see this in study after study. Young people spend so much more time on their phones and they spend with friends at parties, driving, going outside, playing sports.

You name it. If it's a physical world activity, it's in decline. If it's a digital world activity, it's in assent. And it's just impossible to ignore the fact that this replacement of the physical world the digital world has coincided with the largest mental health crisis in youth American history. And I'm a little bit concerned about social AI getting too good. You know, you mentioned that AI is already a decent therapist. I think that's great. I think the world could use more therapy. You mentioned that AI is, we've mentioned that AI is as good as 1,000 interns. That's great.

Who couldn't use 1,000 decent interns? But as social technology via AI gets better and better, I am a little bit afraid of the both predictable and unpredictable ways in which it's going to replace certain human activities in a way that makes people feel weird and disconnected from that which makes them human. It's hard to put my finger on what exactly the prediction is other than just to say, you know, watch the movie her. I find it very unlikely that Spike Jones is no Shadamus and like got the future exactly right. but I wonder if there's other ways we can't yet see that social artificial intelligence will make it harder to be a social human. Yeah, I worry about that too.

I actually, I spend a fair bit of time on AI subreddits because I just find it useful to like see what people are talking about and how they're using this stuff. And I do see like a lot of people just saying, they're talking usually about tools like, you know, character AI or replica, which are these kind of social chatbots that can talk with you and adopt personalities. And there are people who have already formed deep emotional connections with chatbots.

And people will say things like, this is the best friend I've ever made. And on one level, you know, one friend is better than no friends, right? There is a loneliness crisis, and I guess I would rather have people, you know, having artificial friends than no friends. there's man, something about that just sends a little shiver down my spine because I do see a world maybe not that far off where people's closest relationships are not with organic entities. The things that they spill their secrets to that they solicit advice from that they are vulnerable with are not other people. It is chatbots. And to me, I just don't, I don't think that will ever be

how I encounter the world, but I think for people who are maybe younger than us, who sort of grow up surrounded by AI, maybe that will just feel natural. And I don't know what we lose in that world, but I can't imagine it's nothing. Kevin Roos, thank you very much. We'll see you soon. Thank you. Thank you so much for listening. Plain English is hosted by me, Derek Thompson, and produced by Devin Manzi. some great news for you all.

As you probably know, we are returning, have returned back to our normal schedule of two pods a week, so be on the lookout for new episodes every Tuesday and Friday.

If you like our podcast, please rate, give us five stars, subscribe wherever you listen, and I'll see you later.