Big Technology Podcast - The Professor Who Required His Students Use ChatGPT In Class — With Ethan Mollick

Episode Date: July 26, 2023

Ethan Mollick is a professor at the University of Pennsylvania’s Wharton business school and writes One Useful Thing on Substack. He is one of the world’s foremost researchers on practical applications of generative AI and is an immensely engaging speaker. Professor Mollick joins Big Technology Podcast for a vibrant discussion of how AI changes schoolwork and office work, covering his decision to make his students use ChatGPT in class. Tune in for a insight-packed interview that will illuminate where the field is heading. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: [e-mail]

An author and a Wharton professor speaks with us about AI's application in schools and the workplace and talks through the latest in artificial intelligence research. My conversation with Professor Ethan Mollock coming up right after this. LinkedIn Presents. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Ethan Mollock is here with us today. He's a professor at the Wharton School. at the University of Pennsylvania.

He writes one useful thing on Substack. It's a great newsletter. You should subscribe. I've been looking forward to this conversation for a long time. Ethan, welcome to the show. I'm thrilled to be here. Thanks for having me. Great to have you.

I think in terms of the people who have actually gone and used AI and written about AI and its applications in school and at work, I don't think there's anyone as prolific as you, anyone who's built an audience and expertise like you have. So I'm thrilled to be able to be able to. speak with you about this. One of the things that I've noticed is that interest in chat GPT seems to be dropping off your professor. Is that because kids are out of school? Yeah, I mean, there's this sort of stat that everyone is sort of talking about that the numbers are dropped. And it's anyone

of a number of things, right? It's entirely possible that some set of people are losing interest, right? It's not an easy system to use. And if you're using it for fun, you bounce off it pretty quick and move on. It's very possible that it's a big engine for cheating. Schools is out of session. That might be the recalls the drop. The third is there's now actual competition with chat GPT. So maybe people are using Bing or Claude or Bard. So it's hard to kind of make a big jump at this, but I don't think it matters very much because I think you're going to see some churn. Not everyone's going to find this useful right away. That's not really the point. But I have been thinking about it and it's like, okay, not everybody's going to find it useful,

but some people are talking about this as a revolutionary thing that's going to change everything. There's certainly notes of that in your writing in terms of the way that it's going to change education. So how do you square that with the fact that something that's something that can be so revolutionary, can lose some adoption or have the adoption tail off early on. Well, I think, I mean, you have something that is many things at once. AI is a general purpose technology, right? It can do many different things. It can obviously, you know, create content and, you know, cheat on tests, but a general

purpose technology is something that comes around once a generation, think the internet or steam power, and has huge impacts on all aspects of productivity and work. So chat cheptie is just the tight, you know, the, you know, the, you know, it's just. sharpest edge of GPT and general AI use across the many different domains. So if a lot of people are playing with as a consumer, finding it boring and moving on, I don't think that's a big deal because I think the consumer use is the least interesting. I think the question is, okay, what are people adopting this stuff for real work? Are people building wrappers around this and using the API instead? Like, I don't think it meant, like the actual use is going to change

over time. And I think that how many people are downloading the app or playing with the website is probably less important than we have a sort of technology shock overall. Okay, so let's talk about one of those actual uses, which is schoolwork. So, I mean, I visited a university. I went back to Cornell where I went to school sometime, you know, over the past semester. And just kind of was sitting at the lunchroom asking kids, hey, how are you using this stuff? And it really was unbelievable that every single one of them had heard of it, was using it. They were using it in ways that I didn't really anticipate.

They didn't understand concepts that were taught to them in class. So they were going to the LLMs and getting them to teach them in the gaps where the professors weren't able to cover. How have you seen it actually changed the way that education is working so far? And what is the vibe among you and your counterparts about how this is going to, you know, change your jobs in terms of educating students? So I think you're right. I mean, everybody that goes to cheating first.

And that is important and interesting, but in some ways the least important and interesting. We'll deal with the cheating problem, right? There are ways of solving that. They aren't by detecting AI writing, by the way, which is impossible. But there are ways of dealing with the cheating problem. What there, but the really profound thing is, like you said, using this as a universal tutor, using this as a way of creating, you know, of creating new ways to teach, new exercises. I've made AI required all my classes, my entrepreneurship classes.

And like, what an entrepreneurship class have done pretty well? I think out of the intro classes that I know that people teach, people have raised like $2 billion in capital over the last decade or so, you know, plus, like it's, they do a lot really well. Now they're doing even more. I actually ask people to do impossible things. But there's tons of different use cases. So explaining things, explain why you got things wrong, people building demos with it,

getting feedback. You know, it's so it's beyond just cheating. But I also think it fundamentally changes work. People are going to be cheating on essays. People will be cheating on problem assignments. All of homework is threatened, right? We're facing what I call the homework apocalypse. But I think we could build a better world afterwards.

Yeah, there are other interesting applications that I hear that now that I'm thinking about it, there are law students. I guess this might be in the cheating bucket, but it is pretty resourceful law students who are on call in their law school class that are basically chatting with Bing to make sure that they have the answers on specific cases. So how do you deal with that as an educator? I saw the same thing. You put a Harvard Business School case into it.

And if I type into and I just say, tell me what I should say to sound really smart, track the case. It will definitely do that. So it invalidates whole ranges of how we used to do work, right? I don't think we can do assignments the same. way anymore. Now, the plus side is we've known for quite a while that the way we were teaching classes was not the ideal way to do it. The ideal way to teach is actually using what's called active learning and flipped classrooms. That's where you do your learning outside of class and

you do activities and applications inside a class. So lectures are generally a waste of time for the way most people do them compared to actually doing things. So what I hope to see happen is less emphasis on homework as a way of practice and more homes. And more homeschooling. homework, more of the homework outside of class is the initial learning and then the practice happens in class. But that is a radical change from how we used to work, how we used to teach. And people aren't ready for it, right? As you said, like no one's ready for what's about to happen. Essays don't work the same way anymore. Problem sets don't work the same way. In class, conversations don't work the same way. We have to rebuild around these new technologies.

Right. And long term, maybe we will, but short term, as you mentioned, this is really going to be an issue. I'm just going to quote from your piece, the homework apocalypse. So first of all, it really is astonishing to me, A, how much the internet changed the way that people do homework. So you write, one study of 11 years of college courses found that when students did their homework in 2008, it improved test grades for 86% of them, but only helped 45% of students in 2017. And that's because half of the students were looking up homework assignments and answers on the internet in 2017. So if we expand that, doesn't that in the short term, at least, before we go to flip classrooms just totally destroy everything we're trying to do in education.

I mean, people are already cheating. There's 20,000 people, at least as of last year, before Chad ChupD came out, who's in Kenya's full-time job is running essays, right? Like, this has been a longstanding thing. We've just kind of ignored it. So you can't ignore it anymore. And also, a different kind of cheating is enabled. And also, what cheating is change?

Is it cheating if I ask Chatsypti to help me come with an outline, but I don't use that outline. Is it cheating if I ask for feedback on an assignment? Is it cheating if I ask it for 10 different ways to write a sentence when I'm stuck? We have to redefine how cheating works. We have to redefine our plagiarism works. Like this is not an easy sort of first step. So it's a shock. It happened all at once. And as you said, school districts aren't ready. I think in the long term, we emerge much stronger. In the short term, we're going to see people doing all sorts of crazy stuff. I mean, they already are trying to use anti-cheating tools, which don't work. But I expect people to be going back to filling out Blue Book exams as an interim solution.

I expect people to be given as homework assignments where people are scrutinized and quizzed about whether or not they answered things. People will be asked to use Google Docs and teachers will go over line by line to make sure that they are entering them at the right time because you can look at track changes. There will be all kinds of crazy stuff that isn't helpful at first as a stopgap measure until we figure out how to do this right. So I think we'd both agree that like looking things up on the Internet isn't cheating. when you have homework, right?

And you're, I mean, assuming the professor hasn't told them don't look stuff up on the internet for this, right? I think that's normal. And that's already decreased like the, and you put it at the recall. That's cheating. Yeah. If you look up a problem set, if I assign you a problem set, I'm grading you. Okay.

That is a cheat. That's cheating. Right. Yeah, that's cheating. Doing internet research isn't cheating for writing an essay, obviously, but also, you know, but with that homework assignment, it was about looking stuff up on the internet that are exact answers, right? It was about searching, you know, Chegg or whatever.

to find out the right answer to a problem. And that's really become the issue. But you could do that with essays, right, unless you paid someone, which was kind of really cheating. But now you can, right? And the AI will generate essays for you. And by the way,

it has vision. So show it your geometry problem. It'll solve it. Like, that's a pretty interesting world. Are your colleagues coming to you and be like, I mean, obviously they're looking at you as a guy

that understands this. Are they coming to you and they say, I busted a student using chat cheap ET in an essay because here's the word as an AI assistant. and I cannot or I can. What do you think about that? Has that happened? I mean, look, at Wharton, I'm not seeing much bad cheating.

Like, I mean, look, I did have one person at one point during my, when I created, where their assignment switched fonts partway through to the Wikipedia font, and that was a dead giveaway. So there's bad cheating out there. But I actually assigned my students to cheat in class. So it's part of the way of learning what large language models operate. And by the time, what I tell them to do is fake an essay about a personal experience, reflecting back out of personal experience

and applying a class concept to it. Very standard. Reflection is actually a very powerful tool. Applying class knowledge to a real thing is a very powerful tool. So it's a perfectly good assignment. But I asked them to fake it and they have to use at least five different problems

and tell me what happened. And I will tell you, by the time you prompt the AI four or five times with, you know, and there's multiple techniques we can talk about that with prompting. By the time you prompted the AI four or five times, the results are really good.

And no AI detector detects them. And they don't feel like, they don't say as a large language model and they don't end with, in conclusion, and they don't have the chatyPD style lists of things that go on in them because you can make it much more human writing. So go a little bit deeper into this idea behind assigning students to use AI in class. How does that get across some of the learning objectives that you have?

And you think that's going to become more prominent? Well, there's about three different ways of using AI in class, right? So the surface stuff I gave you is that, right? The surface level stuff is you do an AI assignment, you know, focused on a, and learning AI. And that's great. I mean, but, you know, I'm a fairly advanced user of AI. I get to teach my students that, you know, there's a, and so that assignment has a reason. Then the second level of assignment is letting people use AI and taking advantage of strengths and weaknesses. So ask them to do a report and they use AI and they have to critique the AI's answer, right? And that

actually can be very helpful. The third level of AI is integrating AI deeply into everything you do. And that lets you do the impossible. My syllabus, I used to ask people at the beginning of my class, they had to do a class outline where they were, you know, they would outline the business idea they had. Now the outline assignment insists that on top of doing an outline, they have to do at least one impossible thing. If they can't code, they have to code. If they can't do HTML, they have to do HTML. If they can't draw, they have to be drawing. And every assignment they turn in has to be critiqued by at least three famous entrepreneurs through history, which they then use GPT to give them perspective because getting outside perspectives matters a lot. And I expect

to use AI for feedback. And now they do tons of more work than they did before and get far much further than they did before. So there are different levels of embracing AI for a school. So much of the problem with our education system seems to be, and this is speaking from educators, that it is, memorize and spit back. I guess that's why the lecture homework system doesn't work very well because people sit in lecture. They memorize what the professor said. They come to the final and they write that down or they type it down and then spit it back and their ability to retain what they heard is sort of evaluated. Do you think that these things, these, you know, LLMs and the, you know, being integrated into classwork can actually help

create more of the critical thinking and more of the inventiveness that we need to see in students and in the education system moving forward. So I'm going to actually push back in you a little bit. I actually think, so I think we need to learn more facts. And that sounds horrible. Really? It turns out. I'm shocked.

Okay. Go ahead. All right. So here's why. Okay. Because in order to be able to beat large language models or at least to be able to work with them, and who knows how good they're going to get, really. We can talk more about that. You need to be an expert. Like, if you were in the 50th percent tail, that's not good enough.

You need to be an expert in something to be able to monitor the LLM, use it well, be a cyborg that it helps you out with. And we actually know how to build expertise. And unfortunately, there is no shortcut. Like, people want to teach critical thinking as if that's some sort of magical shortcut over what we're doing. It's not. The way you build expertise is unfortunately horribly grinding. It starts with having a deep basis of facts. You need facts. And then you need to start to see the connections between those facts, which is like you move the facts from your short-term memory and your working memory to your long-term memory. Then you need deliberate practice, right? You probably heard the 10,000-hour thing. That's not right. It's not 10,000 hours. But you do need vast amounts of grinding deliberate practice that push you really hard to apply those facts in different ways.

That's how you become a chess master, right? You study a whole bunch of moves and games, and then you play enough with enough against increasingly hard opponents and learn from your mistakes until you get better. So you get good at tennis. That's how you get good at being a carpenter. That's how you're going to get being a professor. So we can't skip the facts phase because that's how humans build expertise. Regurgitating and spitting back facts. So you start to see the connections between, okay, you know, and you're making tons of them here in the room, right, between all of these different topics, you're an expert technology.

That requires you to know a lot of facts about technology that you can pull together and pull, you know, and throw back out in useful ways. Being an interviewer, right, requires knowing a lot of facts. So we can't skip that phase. We can't outsource that. And so in some ways, the AI requires us to actually. double down on the basic knowledge because we can only build the advanced knowledge from the basic knowledge. Interesting. So then what do you think the future of education is going to

look like? I think what makes building expertise hard is that process of deliberate practice is hard. It requires coaching. It requires instruction is mixed in. Like you have to get lessons. It has to increase a difficulty level on a constant basis. It has to be as engaging as possible. You have to be utterly focused on it. These are things that's, It's really hard to do in a classroom. Really hard to do. So we sort of outsource it to a bunch of homework and a lot of grades of doing the same stuff over and over again.

And then you start to work as a junior producer and, you know, in podcasting, and you work your way on, you do crap work. And then you work your way up to mid-level. Then eventually you get good at. Like, we depend on all of these systems to do it. AI might be able to let us skip these things and make expertise easier. A tutor, and by the way, if you look at Khan Academy's Camigo, you can start to see the direction here.

A tutor that actually knows where you need to go and make sure the work is always just hard enough without getting frustrating, that it meets you at your level, that gives you feedback and helps you through the process of deliberate practice. That's an exciting potential future, right? So I think we can get better at building human knowledge and expertise. We know how to do it. We just, the most effective way to teach is what's called direct instruction and one-on-one tutoring, right? Like literally the idea of a one-on-one tutor, people who get one-on-one tutoring acting the 98th percent of a class. Like, it's amazing. But one-on-one tutoring is incredibly expensive. AI can do that. So I think we're looking at a world that unlocks that.

So do you think that this is going to turn to AIs? Well, I think part of the teacher will turn to AI. I think the tutoring instruction will be AI-based. I think you're still going to use classrooms because we need to work with each other. We need to apply our knowledge. We need to get feedback from what mistakes other people are making. So that's that flipped classroom idea. In class, you'll be doing exercises, activities.

Some will be AI-driven, but the teacher will still play a vital role. And they'll be able to provide guidance, support all the other things you need in a classroom. And outside of class, the instruction that used to be delivered through textbooks and through lectures is going to be delivered. delivered through AI. Last question about education. Look, the education system, you know this well, moves extremely slowly. It's not a fast-moving system and everybody inside it, well, not everyone, but almost everyone seems to be extremely resistant to change.

So I'm hearing that, you know, these ideas from you and they seem to make sense. But then if you ask me, like, realistically on what time horizon this stuff is going to happen, 20 years? I mean, what do you think the actual speed of change that is realistic for us to expect? Well, that goes back to that dropout on GPT downloads that you saw before, right? Systems are slow, right? And people overestimate how quickly change happens, but systems resist change because it's not just one thing.

You can't just make one thing different because the rest of the system needs to operate too. The same thing will happen with jobs, by the way. So the education system is slow. It's slow for a reason because there's a lot of interlocking pieces. Like, you can't just advance things as one teacher because you need to fit. into a class system and you need to be able to fit into getting created. The kids need to get able to get to college and you need to know what they're doing and hold them accountable.

Like there's reasons for this grinding slowness. It's not just bureaucracy, right? It's that too, but it's all these interlocking systems in place. And it has to fit into teacher tenure and what parents expect. There's lots of pieces, curriculum. So those are going to take a very long time to change. I think that you'd be surprised at how quickly people can move when it's radical. I mean, schools moved imperfectly online with two days notice. It's insane. Like, I think COVID showed that we can adapt quickly we need to. But I also think it overestimates the need for systemic change and how much systemic

change needs to be just in U.S. school systems, right? The exciting thing about something like Bing is it's available 169 countries around the world. If you are in Botswana right now, you, you know, you have access to the same, same GPT4 that you can get access to as, you know, black rock. Like, there's no difference. And if I give you the right two paragraphs or prompt, you can learn something. new. So I think that we're underestiming the global change on this. I think you're right.

Things aren't going to change overnight, but I think there is a capacity to do it. Does that mean all these predictions? And so let's shift to the workplace now. I mean, it does take time to do this. And workplaces are also slow to change, even though there might be a little bit more nimble than, you know, educational institutions or government. So you think that because it takes this amount of time to adapt and change that all these rumors, all these, you know, this panic, like, oh, AI is going to take my job, might be a little bit overblown right now. So I don't, I think that's part of the reason. So to talk about AI taking my job, we need to get a little bit academic, which is we don't

like to think about jobs in academia. I mean, we do. Obviously, we have jobs. We have jobs. Right. We think about the tasks, the bundle of work that you do. And we think about the systems that your work is in. Right.

And so we can talk more about tasks a second. And it seems like you want to have that, you know, we'll talk about that. But it does matter, right, that you're part of a system. And that does kind of, but on the other hand, task adoption is very quickly, you know, you can easily change a task. And because individuals are incentivized to do action, and because the productivity performance impacts are so potentially huge, like 30 to 80% performance improvements in some tasks, that creates a huge incentive for companies to shift quickly.

So I think we'll see faster movement. I'm already seeing companies do faster things. Part of it is that the APIs that you use to access GBT4 are kind of so easy to do that you could kind of roll your own solution remarkably quickly. Yeah, one of the things that really surprised me in reading your writing is that it seems like this change is actually being driven more by the individuals than the companies themselves. And there are people, you did a poll asking if you use AI at work, do you tell anybody? And 50% of people were like, no. So what do you make of this idea that this is being driven by, you know, the bottom up versus the top down?

In fact, many companies are banning AI, which is quite interesting. And if you talk to people at the companies that band AI, they're all bringing their phones to work and doing all the work on their phones and then emailing themselves, right? I mean, you know, let's go back to cheating, right? Like, in school, cheating is bad. Shortcuts are bad. At work, if you can figure out a way to do all your work and 10% of the time, you are going to do that. Humans are exquisitely built to respond to economic incentives. So if I set it up so that you can do more work and less time and less effort and you can outsource boring tasks, you will do that very quickly. And so we're seeing people experiment all the time.

experimenting with companies is expensive. Experimenting with your own jobs is super cheap, right? You try chat. It doesn't work. Try it again. It doesn't work. Maybe you give up. You try it starts to be interested.

You keep going. The trial and error is cheap for your own tasks, expensive for other people's. So it's very easy to adopt the individual level. The benefits are huge individually, as long as no one knows you're using it. And I talk to people all the time who I source 90% of the work and just don't tell you about it. What happens when people find out? well, so I think that's the real question. If you're a smart company, you're going to be working

really hard to make sure the answer to that is they get richly rewarded, right? Because what I want to do is find out what everybody's been experimenting on. All these companies are used to doing experiments, to doing change to the top down, as you said. They're used to having a boss do things. They have an innovation group. They hire McKinsey. That's not the way of this works. There's no reason McKinsey knows more about how to automate some mid-level manager's job than they do. So I need to get that middle-level manager willing to talk about what they're doing to share that information with me. And to do that, I need to make sure that middle-manager doesn't feel like they're going to get fired. I need to make sure that that middle-level manager knows they'll get

rewarded. Then I won't fire other people because they've ratted out this capability. I want to make sure that people feel incentivized to be part of the team. So this is where having a hostile work environment where you hate the boss but are grinding away work, I'm never going to share anything. If I'm for a company where I'm all in on that company, maybe that's a little bit different. right and so it sort of goes to this question of like all right so what's going to happen to my job it's like actually if you can take these tools and make yourself more productive you become more valuable to the company and i think that like one of the companies really never talk about like ah like you know we wish we did the same thing with less people they always talk about we wish we did more we just have a labor constraint

or a cost constraint and this could potentially flip that is that how you see it well i mean there's lots of different constraints right like what you might be lots of companies also fire people, right? Because, I mean, let's be realistic. Like, if you get a productivity gain, that's what people tend to do, right? Is like, or they view that there's overhead, they slash it. But if you can figure out ways to turn that extra overhang of people in productivity into positive growth, your company would be much more successful, right? And, you know, if you're firing people left and right, when they become more efficient, they're going to stop becoming more efficient because they know what happens. So it's a, you have to

embrace the idea that there might be some short-term inefficiency. to get this long-term gain. But it does tweak the employment contract just a bit, don't you think? Because you get paid to do a job. The contract is basically you're doing this job. This is what I think, you know, someone on your level is able to do. Then you find a way to be, you know, much more efficient. And then all of a sudden it's like, you know, it really does, you know,

basically as the technology has shifted everything there. And it's no longer like it's almost a, I don't want to say dishonest, but it's a completely different way of approach. the job if someone can do it like shouldn't shouldn't i mean yeah okay you're shaking your head no i'm agreeing with you actually i i think i think i i think that you're right that this is completely shattering but every i mean this is gpte general purpose technology breaks things right like this is like old systems are going to get broken everywhere and we have to reconsider them what does it mean to work with an a i like what how much by a free agent working with an

a i to help a company how much is my employment contract matter if i get all my work done in an hour and a half, do I get the extra seven and a half hours on me, you know, seven hours on my own, like, or six and a half hours on my own? If you're a banker, 16 hours on my own, like what happens, right? We don't know. What happens when, like, a lot of our systems at work are built around human limitations. They're built around enforcing limits on human, like, keeping everyone operating at the same pace, making sure you're gear in the machine. What happens when that breaks? We don't know. Like, this is, you know, it's very funny. People talk about the singularity, right, is this thing that is like the AI gets smarter and that

murders us all. And I'm definitely think, you know, that seems worth worrying about. But it's also, the original meaning is it's like a mathematical point that we can't predict what happens afterwards. That's already going to happen in work. I don't know 100% what the future of work is going to look like. I don't know 100% what the future education looks like because we're just assuming that everything stays the way it is, right? What happens is work gets more and more easy to automate with AI? What kinds of categories of work changed? Like, we're barely scratching the surface. Let's go through a scenario. that sort of tends to what you've been discussing.

So I am a manager. Somebody comes to me and says, listen, I read Ethan Substack and I'm confessing. I'm using AI. You shouldn't fire me. Here's the article. This is why.

By the way, I'm doing 100% of my work in 10% of the time. So does the, what is the next step for the manager there? Do they say, we actually need you to do 10 times the work in 100% of the time? Or do they say, yep,

that's good. Teach us how to do it. And we're going to have basically give everybody put everybody on the two-day work week. So there's a whole bunch of options, right? I mean, no, I'm serious. So one option is something we already know works really well, which is let's job craft this.

People are more motivated and do better jobs when they go through job craft, which means working together with manager to figure what their job is. So you can say, hey, what outsource stuff have you done? The 10% that's left, do you find that engaging? Is that what you really want to do? Because a lot of the early state work of AI

is about freeing us from drudgery, which is kind of the good. It's ominous in the longer term, but the short term, it looks pretty good, right? Like, my job is a bundle of tasks that includes many things like fill out expense reports. If AI does that, yes. Like, that's the best thing in the universe. I hate expense reports, right?

And so you could ask this person, what if you outsource? How do we get you to do more of that 10%? Maybe we're giving you a raise because you're doing 10 times the work now. Maybe we're giving you a million dollar bonus because we can spread that idea across the entire company. We're saving that much. Extravagant rewards, rebalancing work towards what people want to do, right? Thinking about what systems we can use to support that person. part of a conversation. Exactly. It's interesting. So I know, I know we kind of danced around

this question earlier about, about jobs, but I mean, like, so we don't know exactly what's going to happen there, but there has to be some sort of impact, don't you think? I mean, there's going to be, right? I mean, and again, general purpose technology, it's going to be a huge, like a huge generalized impact, right? It's going to affect everything in all kinds of different ways. So some jobs are going to change and disappear, right? And like when the telephone system was moved to a digital telephone system, it resulted in a giant wave of change. A lot of people lost their jobs. A lot of things ended up happening very quickly. But most times when technology change happens, people get better jobs and higher paying jobs and the nature of work changes. But we tend to look at the back on those

historically as not in the moment, which is where we're going through now. So there's going to be disruption everywhere in the nature of jobs. Yeah. And it's also, there's something that you hinted at that I think we should expand upon, which is that taking away the drudgery is really nice in the short term, but ominous in the long term. I mean, after reading your work, I didn't really fully think about how this could potentially change like the nature of meaning that people get out out of their work. And you had this one line that's just amazing. You know, you're talking about, I think, letters of recommendations. And you start to say that, you know, we can create documents mostly with AI that get sent to AI powered inboxes. where the recipients respond mostly with AI. Even worse, we still create the reports by hand, but realize that no human is actually reading them. We can view the destruction of busy work as freeing, and we do not yet have to start our, oh,

this is about setting our time on fire as a signal. But let's go back to this idea of the AI's community. I mean, this is one of these things where, like, you know, it seems like maybe one day instead of us getting on the podcast, my AI will interview your AI and will be listened to by AI users who will summarize it for people. What does that change our society? How does that change work? I mean, we have to reconstruct meaning, right? Like, this is the biggest crisis. I think that everything else is secondary. I am sending a report. As a mail manager, the report that I would

write about my employee's work, even if no one reads it, was valuable because it means that I checked the work, right? And that my boss looked at the, looked at this, you know, it saw that I was doing this and is like, okay, the systems of monitoring are in place. Now I hit a button in Google Docs. It creates a automated report that looks plausible. I send it over Gmail with a fake message to my boss, who then hits on a reply and gets a message back. We've just taken the intellectual meaning, but also the reason for doing this work, and it's disappeared entirely. Like, we have to reconstruct this. This means our work systems were built. The reason we have org charts is because in the 1850s, somebody wanted to make sure we had org charts, early railroad barons, wanted to build

a structure that would let them control vast geographic train networks from the top. So they built systems that look like train networks that do that. Henry Ford realized that people, he could hire, if he hired low-paid employees, he couldn't monitor and control them in what they were doing, but if they did one task over and over again, he could keep an eye on it. So thus, assembly lines. Agile development is all about the idea of that we could track bugs and change at the internet, but we still need stand-up meetings to coordinate work. All of those things are going to change as a result of AI being able to coordinate to interact to change, you know, and that's going to change work. The companies that figure this out first are

going to win. It's almost a chicken and egg thing, right, where it's like, okay, the AI is changing work, but it's also commentary on how much of our work is meaningless, don't you think? I mean, you, in your essay, write about how, and this is what I was getting at with setting the time on fire, which is a concept that you've talked about. But basically, people are asked to, like, write letters of recommendation, not because they're actually going to read it, but just to kind of show that the professor would put the time in to recommend the student, and it's a signal. And it just seems that if you can automate this and have AI talking to AI, like so much of the system of work today is kind of garbage, don't you think?

Yeah, I mean, it made, I mean, look, work is broken in lots of ways. Literally, as a management scholar, one of the dominant methods of understanding how organizations work is called the Garbage Pail method, which is just like stuff happens, right? Like, you'll meet each other. It all gets tossed together and stuff happens. AI can help a lot, right? It can help free us for drudgery from bureaucracy. But there's a lot of people who work in bureaucracy.

There's a lot of people who, you know, and there are reasons part of those systems exist and part of them just grew that way. And, you know, and we have to reconstruct all of how that works. As you were saying, the problem with the letter of recommendation is not just that I can push a button and write it, but that the letter recognition that I push the button and write is going to be better than a letter of recommendation I spend an hour writing. because the AI will read all the documentation, all the material, and be able to produce something that's much better.

So the problem is I turn in the old-fashioned school, you know, like properly done, morally correct letter of recommendation. And I'm actually hurting my students' chance of getting the job compared to pushing the button. And we don't have jobs, things built around that. I mean, do I just said the prompt I would have said? Like where I say, I would have told the AI, I like this person their good job, give them a good recommendation. Or what do I do?

We don't know. We don't, but that's one minor system out of so many that's going to be broken. It does seem, so there's been this question of like whether AI will automate all of our work and I can keep coming back to it. But it does, it does seem at a certain point, like the further we get into this, we could effectively just like set it and forget it with our economy. And maybe I'm kind of delusional, but it does seem like we'll still make work for ourselves. Like, why don't we just like have the robots kind of take care of our basic needs and just kind of live? Well, I mean, but that, of course, is the bet, right? We do, like, we're coming back to the billion-dollar question, the trillion-dollar question, actually, I guess,

or what are the companies, what, like $7 trillion, 12 trillion? I don't remember off the top of my head. That's the question, right? The question is, first of all, how good does this get? Right. Like, there's too much obsession. Not that it's wrong about, like, whether we build artificial general intelligence and a machine god, totally get it.

I'm glad people are worried about it. They absolutely should be. Government should be worried about it. We should all have concerns about, you know, whether the machine god will save or kill us. But, like, there's a lot of steps between there and now. And the real salient question for technology is, is this as good as it's going to get? I think the answer is probably no. Will change be linear or exponential in the future?

That we don't know. If change is exponential, then maybe the robots take care of everything is something that happens in the next five or ten years. If not, if it just keeps getting better, we have a gradual squeeze. If you're not in the top 10% of workers, you know, in your task, you're probably, you know, AI will replace you. Then top 8%, top 7%. So I think the question about,

what happens with our leisure time, what happens with work. Depends, like, right now, if AI would pause where it is today, I think it would be 10 years of us absorbing the impact, but I think it would be fine, right? Like, it clearly needs oversight. It does not, there are very few categories of workers that completely replaces. It tends to do drudge work, right? I think would be fine.

Now, what happens if it gets 10 times better? I don't know. Then we start to think about how work gets automated changed in much more profound ways. We have abundant resources on this planet and in the modern world that we've developed, but we still fight over them. Like, it just seems so ridiculous in some ways that we're getting this technology and we're still going to fight over them. I mean, it's going to help.

I mean, the question is whether it makes us fight worse, you know, is the, is a real question. Right. Right. I mean, look, it's hard. When new technologies come around, everyone gets utopian and, uh, and cataclysmic at the same time, right? And so this is the, like, but this is something. something quantifiably different. The question is just whether it keeps going, right? Where we have it right now, it is a complete change in technology, and it's very exciting, and it will be a huge

revolution on par with the internet or maybe steam power. But if it gets 20 times better, then it starts to get really weird, right? Because then we actually have, you know, and if it starts to be able to outperform humans at every task, that starts to get very strange. And we don't know what happens then. We don't know what the limits are, and we don't know the outcomes. And that's where, like, the entire sort of system modifies. Right, because if you talk about, like, all right, the technology gets good enough that you only need the top 10% of workers and 90% you don't need, well, you still need workers. You're not at, like, maximum production.

And that drives inequality in a society and destabilizes it. That's what you mean by the system shifting. Potentially, right? I mean, at the very least, you need government intervention to make things happen. But, you know, again, if we look at the history of work, change will be slower than we think. generally. And people generally tend to get better jobs. We just don't know if this is the end of those rules, right? The historical rule has been technological change. Every other time it's happened has resulted in generally an increase in, you know, in GDP, an increase in quality

of work. There's been some exceptions. Will that keep happening is the big question. Ethan Malik is here with us. He is a Wharton professor and he writes the one useful thing newsletter on Substack on the back end of this break. We're going to talk about a few new models and a bit of a lightning round and take some questions that popped up on Twitter and threads over the past couple days. All right back right after this. Hey, everyone. Let me tell you about The Hustle Daily Show, a podcast filled with business, tech news, and original stories to keep you in the loop on what's trending. More than two million professionals read The Hustle's daily email for its irreverent and informative

takes on business and tech news. Now, they have a daily podcast called The Hustle Daily show, where their team of writers break down the biggest business headlines in 15 minutes or less, and explain why you should care about them. So, search for The Hustled Daily Show and your favorite podcast app, like the one you're using right now. And we're back here with Ethan Mollock. He's a professor at the Wharton School of University of Pennsylvania. He writes the great one useful thing on Substack. It's a great newsletter. We've been talking a lot about his most recent pieces here. Let's do a quick lightning round. So, Ethan, you've been experimenting with some of the bots that have been coming out from the non-open AI, non-Microsoft groups.

So what is your, so I'm going to ask you about your experience with two of them, first of all. So what is your experience with Claude from Anthropic and how do you find it different from OpenAI? So Claude was developed by people at OpenAI, and they said they left because they were worried about open AI's risks they was taking with his AI. So it's supposed to be a friendlier and less harmful AI. They just released Claude Version 2, which, is somewhere between GPD 3.5 and GPD 4. So it's a really good model, probably the second best model out there right now. And one of its really great capabilities that it has in the short term over anyone else is it has 100,000 token context window, which basically means it can

hold a book and a short book in memory. And that means you can upload PDFs to it, and it's very good at working with documents. So it's very good at summarizing and annotating at combining documents. I find it very useful for those purposes. Because it's safer, it also is a little bit preachier, so it can be a little bit more annoying to work with. But it is, it's a very powerful model. And if you're working with documents, I'd strongly recommend trying it. Yeah, you have this chart of breaking down all the different models. And I think for Claude is like, good, but a bit too preachy. Exactly. They can be that way. Well, and I don't really mind some of the others, because without guardrails, AI gets really bad really quickly. But Claude could get a little over the top.

You mentioned that anthropic researchers are dreading what they're building, which is, which is really strange, right? It's like they're going ahead. This is covered in the New York Times article, but they're going ahead and they're really freaking out about the nature of AI. And this is a question I ask sometimes, but it's just so unbelievable to me that some of the people with the biggest concerns about AI are like, man, this stuff can really destroy society. Let's build the next new model. What's happening there? I've got, there are three theories, right? Theory number one is it's all cynicism. I don't think that's the case. I mean, some people are being cynical on marketing it, right? Or paying, you know, paying lips, you know, like, but

but that's one option. The second is that they are genuinely worried, but not that worried, right? So they're not, like, they have to say that they're, you know, have believe in possible doom, but they really think it's going to be okay, which, you know, a lot of researchers do. And a lot of researchers are also very worried. The third option is to take what they're saying at face value, which is that they actually think that they are built, that if they don't build a safe version, people will build a dangerous version. That's always a little bit weird to start an arm's race that, like, if we don't burn down the village first, someone else will burn it down, is sort of,

is sort of a disturbing viewpoint, but that may be a serious kind of approach, right? But they may think that they could shape the future that way. So it's some combination of those things, right? I do believe talking to people at AI labs, many of them are true believers. They really believe they're building, you know, super intelligence. Whether or not they're right or wrong, I don't think anyone really knows. What do you think about this pie bot? It's a Reid Hoffman's bot, right?

It's supposed to be a very friendly bot. I started talking to it a little bit. It can even like play like, it gives you an option to be. like a therapy role, which was interesting. I was like, I'm about to convince, confess my deep, dark secrets. And it's like, your secrets are safe with me. What's up with pie? Pie has gone from driving me absolutely insane to intriguing me, right? So like, it is, it vomits emojis. It tries to be a friend really hard. It will not do work to save your life, to save its life, which is something I push really hard on, you know. But on the other hand,

it is pretty good at chat. Like, it's pretty good at adapting to you in the way AIs are. Pretty good to keep a chat going. If you have not. tried the app, I strongly recommend it because there's ability to get on a call with it, which is near real-time conversation back and forth, which is pretty nuts. So I admire the attempt to create AI that doesn't do the thing the other chat bots do. Is there a market for? Is it an open question? Is it like, is this good or bad for the world? Like high engagement bots are kind of a high risk also with their own weird way. So like it can be friendly, but it takes away time. But I've been increasingly impressed by its ability to be interesting to talk

do at the very minimum. Yeah, it's definitely fun to speak with. Do you think we need all these? I mean, there's so many. There's chat GPT. There's Claude, Bingbot, the Pi, character AI. Like, this stuff is going to consolidate eventually. It has to, right? I don't know. I don't know. I mean, I think the models could get specialized too, right? I mean, I don't, I mean, if we're talking about, you know, electricity providers or, you know, railway systems, like having more than, you know, I don't know how much consolidation there's. And I actually don't think for large language models, there aren't that many contenders, right? And the question, of course, is, is there a flywheel? Once you've built a large language model, right, does that

LLM, like, give you an advantage, right? Does that help you code the next model? And so there's only a few companies that have significantly big large language models so far, right? And so OpenAI, which has Microsoft stuff built on top of it, right? There is Claude and Anthropic. There is the company behind Pi, and there's Google with its bard, right? Those are the models. Like, you know, Elon Musk's announced a new model he's building, but it's going to be years of training to get that thing up and running. So the question is, are there really going to be that many? And will they specialize the way Pi is the conversation bot? AI is the document, you know, Claude is the document bot, and GPT is your general purpose tool. Right. Yeah, that's interesting. That's an interesting

possibility. Do you think that prompt engineering is going to be a new job? I think that's kind of a ridiculous idea. I think everybody's going to learn how to do this. What's your perspective? I agree with you. And not only is everyone going to learn to do it, it's just going to be do it for you. If you already use something like, I mean, go back to pie, you never prompt it once. It keeps a conversation going. If you ask GPT, what I should ask, it will tell you what to ask. If you use AI, you know, art system like mid journey, it went from like when I did mid journey prompts you know, a year ago, those prompts would be these elaborate invocation spells, Kodafrome, Art Station 7.

Like, we just didn't know we were doing, throw everything into it to get a good picture. And now you just say, show me the thing. And it shows you the thing. Like, I think that prompting is going to get easier. I don't know anyone who works in an AI lab who doesn't think that prompting is just going to get easier to the point where prompt engineering is kind of silly. Yeah, I agree. What did the U.S. AI efforts look like in comparison to China?

As an academic who studies this, I'm curious, what you think. I'm not an expert on Chinese AI policy. The U.S. is typical U.S. stuff, which is laissez-faire while a thousand different court cases resolve themselves and there are Senate hearings. It is going to be a slow process to figure out what all this means. Japan, for example, has said pre-training. The training data is, can it be protected by copyright, which is interesting. The U.S. hasn't taken position on this. China, the early evidence seems to be a desire to regulate these systems. But on the other hand, you know, national militaries are probably spinning up these. So I think regulatory systems are going to be really interesting. I think

Europe is really being very careful and we'll see what happens with that. I mean, I think there's a lot of change happening all at once. Code interpreter is just something that you've picked up on and plug different plugins that can help you make sense of data. What's your perspective on those? I think that that's some of the most exciting stuff out there, right? Because it represents, yeah. So code interpreter lets an AI, let's GPT, both take in data, down. give you data to download and also run its own Python. It gives it a Jupyter notebook essentially to do its own work in. And it turns out increasing evidence suggests that when you give AI tools, it becomes much

more capable. So given the ability to use tools, it first of all solves all the problems that have made AI really annoying to work with, right? So if you've ever tried to use AI to work with language, like that is a hard thing to do, right? It doesn't understand sentences and paragraphs the way we do, right? If you ask you to count the number of words in a sentence, it can't. do that. But code interpreter will write a little program to do that. It'll solve math problems that it couldn't solve. I write a little program. And it starts to really complex stuff.

Like it will, you throw data at it and it will apply theory. And it will, I give it a paper, I give it, you know, I did a little experiment where I gave it the NBA playoff data and had it find interesting stuff. And it found all these cool hypotheses and then graphed it. And then I threw in some work by, by Tufti, who's a famous graphic design, infographic design person and said, apply these skills to make the graph better. And it did. Like, there's a really powerful democratization of coding and analytics that is already starting to happen.

Do you have to know how to code to use them? I don't. I can't code in Python to save my life. But I do know stats. So I can look at the statistical outcomes. But no, I don't need to know code to do it. We got some questions from social media. So one person asked, how is accuracy necessary in terms of, like, let me just read the actual

question. Taka, ask him about the necessity of accuracy, whether it's a core need for enterprise use cases for LLMs. So should tools like code interpreter, here we go again, give companies greater confidence that LMs won't hallucinate as often and provide reliably accurate outputs? So this idea of hallucination is a real problem, right? Yeah. I would argue that it's probably not as big a problem as people think. So first of all, hallucination rates are dropping. So we don't have a lot of comparative data, but there was a really cool study where they gave the variety of AIs, the neuroscience board qualifying exam, neurosurgery board qualifying exam.

And unsurprisingly the GPT pass the flying colors, but they track the hallucination rates. Bard hallucinated 44% of the answers, GPT 3.5, chat GPT, 22%, and GPT4, only 2%. So hallucination rates do seem to be dropping. And then when you attach to a code interpreter or data or PDF, it drops further. But hallucination is a genuine problem, which is why right now there's a huge advantage in that expertise thing we talked about. If I'm an expert, I can check over the results and fix problems, and I do need to do that with code interpreter. So I think that, you know, that it's less of an issue. It favors experts. And also, there's a lot of jobs where, like, accuracy is the number

one thing. If you're doing marketing writing, it's not that big a deal. Even if you're doing customer service, a lot of the interaction, if there's small errors, they don't make as much of a difference is getting the big things right. So the other question is how accurate you need to be. For some applications, AI is totally out because it elucidates. But for a lot more than you think, it works. There's some concern that people are, we're going to develop brain chips with AI inside them and attach them to our brains. And we could like have our data sucked out and stuff like that.

What do you think about that? That feels that in a world where we're living in science fiction, that still feels too science fictional right now. So I don't think involuntary brain chips are in the cards. And physical world and biology are much harder than. software. So I think let's not get ahead of ourselves too much, I think. Would you get a brain chip if you could? No. I mean, no one's getting generation one brain chip. I mean, you know, and also like, you know, I think that, and I think when you look at what the actual brain development,

you know, the chip stuff is it's not telepathically communicating with AI. I think we need to be really careful about, you know, even a world where a lot of hype is coming true, there's still hucksterism going on. So there is no telepathic brain chip in the near future. We don't know how that, you know, we don't know how LLMs work. We don't know how humans' brains work either. Like, it's, we know how they work, LLMs work technically, but we don't know all the details of, you know, why a particular decision's made.

I think we're going to be a long ways off from connecting AI directly to our brains. Yeah. Are there models outside of LLMs that you're looking at or interested in? I mean, so LLMs underlie a bunch of other things, right? So large language model and the transformer technology that powers it is what is powering art-based AI, dialogue, and, you know, podcast creating AIs. you know, all of these things. So like the large language model technology is the transformer technology

and attention mechanism that's at the heart of this technology are what's driving all of these sets of tech changes all at once. Okay. Last question for you. What are you looking forward to in the fall when students come back and how are things going to be different in terms of the way that you teach? Because you basically had to deal with this mid-stream this year. What does next year look like? I'm excited.

I mean, I think like it's great to take the burden. Like, it has allowed me to reimagine how teaching works in really exciting ways that I think will benefit my students and make things interesting. But that's for me. I think a lot of other people are terrified and or putting their heads in the sand or they try AI a bit or they've heard of AI. I mean, I hear talk to you all the time. We've heard of it, but I haven't really tried it. Like, I think that we're in for sort of a mess. Yeah.

Okay. I can't wait to watch. I hope we can keep in touch. Ethan Mollick. Thanks so much for joining. Thanks for having me. Thanks for being here. Thank you, everybody for listening.

Thank you, Nate Gwattany for handling the audio. LinkedIn for having me as part of your. podcast network and all you the listeners. If you enjoyed, please hit five stars on Spotify our podcast. First time here. If you hit subscribe, that would be awesome. Thanks again to Ethan. I've been looking forward to this conversation for a long time and it delivered all the way through. So I really appreciate you being here. All right, we'll see you next time on Big Technology Podcast.