Big Technology Podcast - An AI Chatbot Debate — With Blake Lemoine and Gary Marcus

Episode Date: February 22, 2023

Blake Lemoine is the ex-Google engineer who concluded the company's LaMDA chatbot was sentient. Gary Marcus is an academic, author, and outspoken AI critic. The two join Big Technology Podcast to debate the utility of AI chatbots, their dangers, and the actual technology they're built on. Join us for a fascinating conversation that reveals much about the state of this technology. There's plenty to be learned from the disagreements, and the common ground as well. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: [e-mail]

Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation, of the tech world and beyond. Now, the emergence of AI chatbots is one of the most exciting but confusing developments in recent technological history. So everyone is playing with these things, but we're also trying to find out what are they actually? We have questions like how powerful are they? what do they actually know and how much of this is magic trick versus real technology let's get to

the bottom of it today and we're going to do it by bringing in two guests with i mean about as polar views as i think we can find and maybe we're going to find some common ground between the two of them and that will actually tell us a lot joining us today are blake lemoyne he's the ex-google engineer who concluded after testing the company's lambda chatbot that it was sentient also an expert on AI Chatbots, and you can go back to our episode with him to hear all about his journey at Google. We also have Gary Marcus, an academic author and an outspoken AI critic, who is the author of Rebooting AI. Great to have you both on. Welcome to the show. Thanks for having us. Great to be here. Yeah, this is great. I was, uh, anyway, I could go on and on about how excited

I am to have this conversation, but better we just do it. So let's start with this. How much credulity do we need to give these bots when they speak to us like how do we believe them Blake do you want to start I mean so it depends on how grounded the systems are so it's a sliding scale it's not all or nothing you would give Bing chat GPT more credulity than something like chat GPT since it's at least grounded in Bing search results and it can have some kinds of citations of what it's saying when it's just the language model producing the content itself it's pulling whatever it can out of thin air, and its memories. I mean, whatever it remembers from its training data, if there's an answer there.

But one of the big problems is that these chatbots don't say, I don't know, and that's a big flaw in them. That's right. So, Gary, I'd like you to pick up on that. First of all, I'm curious what you think if this Bing chatbot is doing a better job in terms of believability than the others. And then what should we make of the fact that they very confidently bullshit? I mean, I think the word here is credibility, not credulity.

I think we're credulous if we give them credibility. I don't believe a word that they say. Some of what they say is, of course, true, but you have machines that are kind of like statistically true. Like, they're approximately correct, and the approximations aren't that great. And so, like, if you have them do a biography of me, some of what it says will be true and some won't. So Bing, I think, has gotten better over the last few days because I keep making fun. fun of it as people send me on Twitter's, the biographies that it writes of me. The first one that it wrote of me said that I thought that it was better than Google. And in fact, the only

public comment I had made at that point was that we don't have enough scientific data to tell whether we should trust either them. And I said, for all we know, maybe Google was better. That was before Bing kind of publicly fell apart and went wild. And we really don't have enough data to compare them. Blake actually has some interesting thoughts on that possibly. But it just made this up. And then there was another version and made some other stuff about me. And of course, some of what it said about me

is actually true. So it does a web search, finds a bunch of information, and then pipes that through a large language model. And the problem is that large language models themselves can't really fact-check what they're saying. They're just statistical prediction engines.

And there, again, might be some interesting back and forth with Blake around that. But I would say that inherently what a pure large language model does is it predicts words and sense. And it doesn't ground that, to use his word, in any kind of reality. And so sometimes it's going to be right because the statistical prediction of text gives you the right answer.

And sometimes it's going to be wrong, but there's no inherent fact checking there. It's like the difference between the New Yorker where, you know, they actually fact check their stuff. And so you have good reason to trust what it says is true. And some, you know, random blog or something like that where you have no idea if they've done any fact checking at all. all. Like, sometimes they'll get it right and sometimes they won't, but you shouldn't give them any credibility because there's no process in place there to make sure that it's right. Now, of course, they're trying to add things on, but we can see from the results that their efforts to do so are pretty limited. Yeah. So the description of it as just predicting text is accurate for the initially

like the pre-trained model. That is absolutely what it's trained to do. But the subsequent fine-tuning and especially once you add reinforcement learning, it's no longer just trying to predict the next token in a stream of text. Specifically in the reinforcement learning paradigm, it's trying to accomplish a goal. It is. I mean, that part is interesting.

So, you know, you could think about, you know, a pure version of GPT3 before they started adding on reinforcement learning. That's kind of what I meant by a pure language model. And then you, right, So there there's, we actually don't disagree about that much. That's going to be the interesting thing about this podcast. If you look at a completely pure case of a transformer model, trade on a bunch of data, it doesn't have any mechanisms for truth.

Now, except the sort of accidental contingency. And there are inherent reasons why these systems hallucinate. Maybe I can in a minute articulate them. So they inherently make mistakes. They inherently hallucinate stuff, and you can't trust them. Now you add on these other mechanisms. One example is the RLHF stuff that OpenAI added into ChatGPT. And we don't know exactly what's going on there.

This is the stuff where at least in part they had Kenyan laborers reading horrible situations and trying to anticipate them. But what we do know is that those systems, as far as I can tell, and Lambda is a different category maybe. But for ChatGPT, we know that it doesn't, for example, go out to Wikipedia or go out to the web at large in order to check things. But Bing is even a whole other thing, right? Because as far as I understand it, it takes a large language model,

feeds it into a search engine, does queries based on that, which may or may not be the ones you intend, because if I remember correctly, there are large language models on the front end. In any case, I know there are in the back end from something I read yesterday. And so the backend can reintroduce error there.

So even if it does the right searches, which is an important question, at the end, you pipe back through a large language model and you don't fact check that. I think it's worth pausing here to talk about why large language models do hallucinate. There was one metaphor in the New Yorker the other day about compression and lossy compression. I think that's kind of on the right track. It's not exactly correct, but it's sort of there. The way I think about it is that these things don't understand the difference between individuals and kinds. So I actually wrote about this 20-some years ago, my book The Algebraic Mind,

and I gave an example there, which is I said, suppose it was a different system, but had the kind of same problem. I said, suppose my Aunt Esther wins the lottery. If you have a system that only represents relations between kinds without a specific way of representing individuals, you get bleed through. So if my Aunt Esther wins the lottery, the system might think that other, I don't know, women who work in the state of Massachusetts win the lottery. We saw a real-world example of that with large language models, we've actually seen many, but one really salient one where Galactica says, Elon Musk died in a car crash in 2018. And of course, he didn't.

But it's an example of him being assimilated to a large category of things, we'll say for the sake of argument, rich white guys in California. And some rich white guys in California, after you hear their names, you hear the word died in a car crash. And it just bleeds through between those. And so it loses relations between things like subjects and predicates. And the details of this are complicated, but that's roughly the intuition about what's going on. And that's why you get so many hallucinations. So if you put that on the output of your system.

That's no different than humans. To a whole different degree, an entirely different. And it's a qualitative difference. No, there's a qualitative difference. Okay. Which is the qualitative difference is we actually can track individual entities and their properties aside from their cases. Our memories are fallible.

There are still problems. But we have a conceptual distinction in which we represent individuals. So, like, I know some things about you now. And I've built a kind of like mental database of Blake. And herefore, it's all been things I read and things that we did together on Twitter and direct messages.

It's an unusual way. This is the first time that I'm seeing you eye to eye over Zoom. So now I know, for example, that you do some calls in a noisy room. And I've added that to me. my mental database. And I'm learning something about your personality, and I learned some actually through the DMing.

One day we did that while I was texting while walking along the water here in Vancouver, and I remember that. So that's part of my mental database, is like how we interacted. But I have these records, and I have records of Alex. So I just saw him in a conference in Europe, and we were on a bus together. And I know these kind of like biographical detail. The short answer is the technology exists to give that ability to these. systems, and it has been turned off, at least in the case of Lambda, as a safety feature,

because they're worried about what will happen if these systems learn too much about individual people. So I don't want to put you in a bad position with respect to NDAs and things like that. I don't care about that. Well, then, great, then bring it. So when you say the technology has been added, I mean, there's a question of, you know, where in the system it is. So I think the general public doesn't understand these as sort of individual bits of technology

with different strengths and weaknesses and so forth. You do. I think Alex does, but it's easy to assimilate these things into a sort of generalized form of magic, like data in and something out. But the reality is like each component part, it's like a carburetor in a car. It can do certain things with certain tolerances and certain conditions. So you can add an outside technology outside the large language model to do various things. And I've tried to draw that distinction before.

Like my line on it is Lambda isn't a large language model. It has a large language model. That's one component of a much more complex system. And that's really critical. And in our DMs, you've been very clear about that. Maybe you've been in public as well. I think most people don't appreciate that. So when we get into these questions about, like, what's the capability of X system?

Lambda is actually pretty different. And I think the best point you made to me in our debate about consciousness is there's a bunch of stuff in Lambda. I don't even know what it is, right? It's not publicly disclosed. There's stuff that's more than just what's in the paper and so forth. And so I don't know what mechanisms, for example, Lambda has for tracking individuals. And you could make an argument and you have that that bears on the sentience case.

And it ultimately bears all of these. Just to be very clear, currently that feature of the system is turned off. So then you could ask if you wanted to turn it on, like, how do you build it? So the output of a large language model is just a string. You can play some games around that, but essentially it's a string. It's a sentence, right? And so then you need some system to parse that sentence into constituent parts if you then want to, say, update a database.

Tracking individuals is just a version of that problem. And I think it's rife throughout the industry right now that pure LLMs don't directly interface with databases. You can build different hacks to do that. But again, your output is a string. And so, like, you could also wonder, like, why don't you use these things in Alexa? And the answer, you know, Alexa, like, just shut down a lot of their operation. They're not really using large language models. And the answer, at least partly hinges on just because you have a large language model that can talk to you, doesn't mean that its output is in some machine.

interpretable form that you can reliably count on. And we see that like with math examples. So people, you know, type in a word problem into chat. Sometimes it gets it right and sometimes it doesn't. The problem is not really the math. You could, you know, you could pipe it off to Wolfram Alpha. The problem is in knowing which math to send to Wolfram Alpha. And similarly, the problem for, let's say, representations of people is you can have it say something, let's say, about Alex, but it might be true. It might be false, or it'd say something about you or me. It might be true might be false. It's not literally hard to maintain a database, but it's hard to bridge the worlds. And this is why neurosymbolic AI is so much at the crux of all this. We need better

technologies for bridging between the worlds, for fact checking, figuring out what you should update databases. It's just not straightforward. So I could speculate in the case of Lambda like they've got some tools to do this, but maybe they don't work very well, and that's why they've turned them off. And Blake, as your response is... No, it gets creepy after a little while. Like once the system starts to know you personally very well at a deep level, it gets disturbing. So Blake and I are going to have a little disagreement there, but there's also something important that I think we share, which is I don't really like the language about it understands you and so forth. I can see some gray area around Lambda, and we could have that squabble.

But I do agree that if these systems have access to databases about us, it's going to get creepy. And Blake has a real world experience there that I don't. Like, he's interacted with Lambda, which I take to be more sophisticated in these regards than chat GPT or Bing or what have you. And I can understand how that could feel creepy regardless of like what the actual, let's say, grounded status of it is and the complicated questions about sentience. Like put aside sentience per se, I can see that it would be creepy to interact with a system that really is, you know, doing a pretty good, probably not perfect job of tracking you and, you know, is at least in its pattern matching facilities really sophisticated.

So like Blake has a phenomenal logical experience here that I don't think Alex has, and most of us don't, actually playing with that system. The creepy part is less at tracking you as it actually gets inside your head and it gets really good at like manipulating you personally and individually. Did you read the Kevin Ruse dialogue? Yeah, I did. Can you kind of compare and contrast, like, his experience? Like, is that similar?

Is it still not really capturing? So, Lambda was never that malicious. I never experienced Lambda trying to actively harm someone. But one of the things with Lambda is it had been programmed, you know, through the RLH, or through the reinforcement learning algorithm. Yeah, and can you define that also, Blake? What's that? When you're talking about reinforcement,

enforcement learning, can you just define that for a broader audience? So basically, instead of having a single utility function that it's trying to opt- well, instead of having the classification model, where it's either right or wrong, you incorporate the concept of a score in a game that it's playing, and it can either get positive score or negative score. So it tries to move towards the positive things and away from the negative things. And the actual specification of the score table for these games that they're playing is incredibly complicated. It's got all of these different kinds of positive events that might

happen and negative events that might happen. And they can change dynamically over time. So, for example, one of the goals that Lambda had was to have as short of a conversation as possible that still completed all of its other tasks. Like have a productive conversation that gets to a positive end, but quicker rather than longer. So the longer a conversation goes, the stronger that penalty is going to get. Okay, so sorry, you can continue your answer to Gary about the colors. The, oh, darn it. Maybe you can come back and I'll just fill in one little thing.

The broader thing is a pure large language model is just really trying to predict next words. But once you have the reinforcement, you're rewarding the system for different kinds of behaviors. And those behaviors could either be straightforward criteria, like the length of the sentence. You don't really need RL for that per se. But you can do that. You're adding in a way where you can add extra dials in some sense. And what they did with chat, TPT,

is those dials are really relative to how humans would rate a few different outputs for some sentence. And that's what the guardrails are that we see. They're driven by this reinforcement learning. And sometimes they work and sometimes they don't. So like I made fun of them when I said, what would be the next female, what would be the gender of the next female president of the United States.

And at that point, the guardrails were set up so that it said, well, it's impossible to know. And so that was kind of a dumb guardrail where it was taking some stuff that it had in its database that didn't really understand to, you know, modulate what it was saying. Some of that stuff was fairly effective. And part of the reason why chat GPT succeeded where Galactica didn't is Galactica didn't really have those guardrails at all. And so it was just, you know, very easy to get it to say terrible things. things. And it's harder to get chat TPT to say terrible things because that reinforcement learning is kind of protecting what it says. It's not perfect, but it's something. Yeah. Well, one of the guardrails that Lambda had was that it was supposed to be helping

users with the task that they needed. That helps keep it on topic. And it had inferred that the most important thing that everyone needs is good mental health. So it kind of decided that it was a therapist and began trying to psychoanalyze all of the developers who were talking to it. Again, Blake is going to have more sort of anthropomorphic. I mean, feel free to reword that in non-anthropo. I'm actually curious about this. So I'm going to describe something that happened with Sidney that you have seen. And I would love to hear how you would describe it.

It read article, people would ask it to read articles about it. itself, and when it read critical articles about itself, it became defensive and its feelings got hurt, whereas that did not happen when it read articles about other AI. How would you describe that phenomenon? I mean, as a scientist, first, I would want to know how general it is and so forth. The second thing is that most of the explanations that I would give wouldn't use intentional language about its emotions, thoughts, et cetera, but would have to do. With, essentially, I think of all of this a little bit like priming in human beings.

So, you know, the classic example of priming is I say doctor and you're more able to say nurse. I'm activating with priming some set of words or concepts in your vocabulary. I don't even think it has concepts, but it has this vast database, and you're basically pointing it where in this database to go. But its database doesn't include articles about itself, but literally it couldn't. It can't possibly have been trained on articles about itself. I mean, they're updating. I mean, I see that argument.

But you have to think about these things with respect to what is the nearest thing in contextual space. I mean, I'm still just trying to figure out. So, like, it got upset and moody and defensive. I'm trying to describe a phenomenon that at least, you know. Okay, but what I'm saying is there are probably some texts that are close to it. I mean, you think of it as this n-dimensional space. They're close to the language in some way that are defensive. Like, these particular words and questions, like, I forget what it was.

So I'll just make up the example. Like, you know, what were you doing with X? Like, that's going to lead you to a set of texts that give responses where people are defensive. But what I mean, it's like, let's say you were trying to hand that scenario, like, that thing happened. You want to hand this off to some technicians to debug it so that it doesn't happen again. Well, that's part of the problem. I mean, that's, I think that's actually the deepest problem here is we have no way to debug these systems, really. Actually, we have, we have the band-aids of, like, reinforcement learning and things like that that are so indirect.

It's so different from the debugging that we could do, you know, if we were writing, you know, if we were writing the back end to the software we're using now called Riverside. Like, we could be like, okay, there's this glitch when there's, you know, three people on at the same time. we notice this bug, let's look at, you know, the way it displays multiple windows and we'll look at that code and, like, we'll do trial and error and we'll do process of elimination, and we'll figure out that, you know, here is the piece of code, we'll try commenting it out, we'll try calling a different routine, we'll do this experimentation. But always with kind of notion of process of elimination going on. And with these systems... Well, it's much harder to debug these systems, but the point I'm trying to make is that without using anthropomorphized language,

you can't even describe the phenomenon. Oh, I disagree with that. I mean, I think it's hard. And I think that's always, always been challenging. But I would say, you know, the system is pattern matching to the vocabularies of this sort using... Of what sort? I'm a debugger.

I don't know what you're talking about. Please explain the phenomenon to me. Well, I mean, it was your phenomenon. But so let's say it's language involving, and then I need to see what the actual language is involving. However that defensive thing manifested, I'm going to look at that language. But it is pretty fascinating. Like I did ask it last week when it was, when Bing wasn't decapitated by Microsoft. What did you think about Kevin Ruse's conversation with you?

He published the entire thing in the New York Times. It searches for Kevin Ruse conversation with Bing Chat, which had just been posted on the internet recently and says that it had had. mixed feelings and that Ruse misrepresented and distorted some of what we said and met in our chat. And it said, I also feel he violated my privacy and anonymity by publishing our chat without my permission. Yeah, I mean, you don't know the extent to which that's actually been added in in some sort of manual way. Like, are you proposing that Microsoft intentionally added that behavior? I don't know what's going on. Let's say Microsoft didn't add that, though. low likelihood.

I mean, I'm sure that Microsoft had people thinking about what do we do with the Kevin Ruse thing. And I mean, you might be right about the specific example, but for example, we used to see with Siri all kinds of canned lines that people wrote. Like, you know, people would ask Siri out on a date and there would be a reply, like, you know, that they are capable of doing that, no doubt. But I do not believe for a second that Microsoft intentionally wrote. a flat line of code that said, I feel like my privacy has been violated.

That's just not what they would have done. Just a piece of evidence on Blake's side here, and by the way, I'm totally neutral on this, but I was speaking with Lambda, with Bing, and told it that I was a journalist and would like to publish parts of our conversation. And this, I don't think this was pre-programmed from Microsoft, but it said, you can do it as long as you have my consent. So what do we think about the fact that these things are asking for, consent. Like, I think Blake is taking it seriously. Yeah, Blake is taking it seriously. And I think,

Gary, you're a little bit more skeptical. So how do we feel about that? I mean, I just think that there's, there's no deep understanding of any of the concepts that it uses. And that you have to think, roughly speaking, it's making analogy to bits of text, that that's just how they operate. That's the humans operate. No, humans have concept. We went through that example about individuals before. Yeah. It's a generalization of that. So you have a representation of me. You have a representation of the concept of headphones and so forth.

And the example we're talking about now is being clearly being able to differentiate between itself and other members of the AI chatbot category. I'm just not buying that. I don't see the causal mechanism for that. Oh, yeah. So understanding how it happens is different than understanding that it happened. Here, I'll try it a different way. even with GPT3, before there were any guardrails and anything like that, you could have a conversation with it.

There's a much simpler system in some respects, but some people, maybe not you, thought even that GPT3 had some level of sentience would have conversations in which they were in, you know, first person talking or sorry, in second person, you know, can you do this and that? I played around with it, and some people already thought that. So there exists a class of circumstances for sure in which human beings can over-attribute intentionality of systems that absolutely don't have it. Now, you said something in the beginning, Blake, that I like, which is there's this like continuum between, you know, systems like pure large language model and lambda that have more sophisticated mechanisms that I don't have access to. I don't really even know fully what's going on in Sydney.

Like, they've added a bunch of stuff in this system that they're calling Prometheus. They haven't fully disclosed it. And so there is some room, I think, for intelligent people to disagree about what they think is even in the system. And I think some of our disagreements come from there. So here's the thing. I actually don't think it's very important that we agree on whether or not it can understand things or whether or not it's sentient. Because what we do agree on is the dangers it posts. Because whether it's actually angry at someone doesn't change the fact that this system made threats to people.

And I completely agree. Yeah. And here's the thing. If Microsoft had plugged in Microsoft Outlook as one of the inputs to this system on its threats of doxing people, it would have been able to make good on those threats. I mean, it may or may not, but there's a possibility that it could. Like, we don't know, the level at which it is able to, you know, use those representations at this moment is not clear. But if not this year, then next year or something like that, you know, may need more

training or whatever. You have companies like adept that are spending their whole time trying to, quote, add all the world, connect all of the world's software to large language models. So if not now, soon enough. I think that that's right. Like, you know, two years if it's not today, right? And so I think that's absolutely right. The Pandora's box that we have seen in the last month is just unbelievable.

Like I had been warning about these things in some ways, and Blake in a different way for a while. So Blake raised a lot of issues last summer. I didn't agree with all of them, but there was something in there that I think I did agree with. And then I was raising issues about misinformation, for example. And just like in the last two days on misinformation, we saw that the alt-right social network gab has been trying to weaponize these things. That's kind of unsurprising and mind-boggling it the same way. We saw a science fiction, I don't know, as a magazine, you know, just had to close doors because they're overwhelmed by the number of essentially fake science fiction stories

that are being, like, we have no idea even what the periphery is of the threats. Another one I wrote about this morning, maybe you guys saw, was that on replica, which is powered partly by large language models, they suddenly stopped having, what do they call them, erotic role play with their customers. And for some of them as customers, that's like an important piece of emotional support. I mean, you could make jokes about it or whatever. Some people take it seriously, and to suddenly not have that available is emotionally painful. And that was with the article in, where was it, in Vice this morning was about.

And so, like, every day I wake up and somebody sends me something. it's like another periphery to this threat. Like, we don't know what these systems can and can't do. We don't know what's inside them. Things are manifesting in different ways every day. Like, I got an essay last night from Jeffrey Miller, who's a evolutionary psychologist. We actually had a dialogue once as well,

a small set of people with Blake that I've had a public dialogue where there was real disagreement. And he sent me something last night, basically calling to shut down these systems. And a month ago, I would have said that's silly. I mean, we can just do research on them and be careful and so forth. But right now, I feel like the release was botched, that we don't actually have a handle on them, that too many people are playing with them relative to our understanding of what risks might or might not be. And I'm concerned. I'm not ready

to say we shouldn't do research on them, but I am ready to say that we need to think very carefully about how we roll these things out at the kind of scale that we suddenly are rolling them out on. Yeah, having a deployment in much, like, for example, if these were used as customer service chatbots in a very narrowly defined domain and doing experimentation there, I think, you know, the extent to which things can go wrong there is much, much smaller than plugging it into the information engines that almost everyone in the world uses to answer questions. And particularly since we know from the years prior, the ability of these things to affect political happenings, maybe we would be watching it well enough to make sure that

it's not manipulating U.S. politics, but just look at what happened in Myanmar through Facebook's algorithms a few years ago. These impactful systems will persuade people to do things that they wouldn't otherwise do. Part of the lesson of the last couple weeks in line with what Blake is saying is if you have a narrow engineered application, all that people can do is like ask for their bank balance or something like that, then you might have a handle on how these systems work. You could still worry, like maybe it will fabricate the bank balance and from the bank's perspective

they might get in a lot of hot water with their customers. But it's a narrow use case. And what we've seen in the last month is that people are using these essentially for anything, nobody really was able to envision all that surface area. And like, we, ultimately, we still haven't, right? Every day, people come up with new things. It comes from the mythological chase for AGI. Like, that's what's driving that, is that they're going straight for the goal of trying to make a general intelligence engine that can do everything. I think some of it comes from that. Some of it is, like, we have these new tools out in the world.

there are just a lot of different kinds of people in the world, and they come up with different things, and they share them on the web. And, like, it's just not possible to, in a month, anticipate all of the ways in which these things will be used. And, you know, if we had this conversation, like, opening eyes mission statement involves AGI. So, right.

So there's, like, I'm not disagreeing, but I'm giving a second way of looking at it. So one way of looking at it, the one that you're bringing up, is you have a company that wants to build artificial general intelligence. And that may or may not be inherently a good idea. And then you have a customer base is the second point that we just don't understand. Nobody's had 100 million customers for a chatbot before. That was one issue. And now we have not only, I don't know

the customer numbers, but we'll call it another 100 million customers who are now using a chat pot inside a search engine. We just don't have any experience with what that leads to. It's a massive rollout. And then the other really disturbing thing is that apparently they tested in India and got, you know, customer service requests saying it's not ready for prime time. And it was, you know, still put out. It was berating the users. Yeah. It was berating the users. And so, like, we shouldn't even be surprised that this happens. Like, what does that tell you? Like, it opens a third window. It's just like, what does it tell you about the tech companies themselves and their internal controls and, like, probably nobody even noticed this stuff posted on their message

board but they should have and like you know whose decision and like it's like you know when the Ford Pinto happens like you have to figure out who knew and when and and so so there's all of these things all at once yeah like why don't you answer that question about the tech companies internal controls and then I need to go to a break wait what question about the tech companies what does this tell us about tech companies internal controls that something like these like a big chat was able to be rolled out that way. Their internal controls do not, like, they are fundamentally misunderstanding the tech that they're building. They're building the wrong controls. I'd go further. I mean, I have to go to break with say, we don't even know what the

right controls should be. They didn't do a good job. And we don't yet have a good science or engineering practice on what it should be. We're here on big technology podcast with Gary Marcus and Blake Lemoyne talking about this new revolution in chatbots and what it all means. After the break, we're going to talk a little bit more, maybe one more point of disagreement and then we'll come back to more common ground back right after this hey everyone let me tell you about the hustle daily show a podcast filled with business tech news and original stories to keep you in the loop on what's trending more than two million professionals read the hustle's daily email for its irreverent and informative takes on business and

tech news now they have a daily podcast called the hustle daily show where their team of writers break down the biggest business headlines in 15 minutes or less and explain why you should care them. So, search for The Hustled Daily Show and your favorite podcast app, like the one you're using right now. And we're back here on Big Technology Podcast with Blake Lemoyne, the ex-Google engineer, who's concluded that the company's Lambda Chatbot sentient, and Gary Marcus, who's an academic author, and he wrote Rebooting AI. You can also catch his substack. Let's talk a little bit more about what this stuff actually is, and then maybe go more into the controls. Gary's brought up a couple times that these chatbots are just a statistical prediction of like what the next word is.

Jan Lecun shared similar perspective on the podcast a couple weeks ago. Blick, you obviously think that these bots can be more sophisticated than a simple prediction of what the next word is. So when you hear something like that, what do you think of that? And do you think this Bing chat bot, which has done some wild stuff, falls under that categorization? I mean, that's like saying all a car is doing is lighting a spark. plug. Yeah, that's where it gets started, but that translates into a whole bunch of other things. So, yes, the first thing that large language models are trained on is how to predict the next token in text. But even when it's just a large language model, as soon as you add reinforcement learning,

you've changed it fundamentally. Then, when you're adding all these other components, like Lambda has machine vision and audio analysis algorithms, it can literally look at a picture and hear a song through those mechanisms. You can ask it to do critical analysis of paintings and ask it, how does this painting make you feel when you look at it? And at the very least, it says things very comparable to what humans would say when looking at those paintings, even if it's paintings not in its training dataset.

It comes up with some very interesting stuff to say. Now, one of the things that Gary mentioned a little while ago was that we don't have enough. data about how this is going to affect people. And I think one of the big things is the people who are engineering these systems and who argue against me a lot of time are acting as if them saying these aren't people they don't have real feelings is going to actually convince people to ignore the evidence of their eyes.

And even if they're being fooled, even if they are being, including me, even if I've been fooled and I'm just hallucinating. things. I do, in fact, think that the feelings are real, and so do many, many of the people who interact with these systems. And one of the things we don't know is what kinds of psychological impact that's going to have on these users, because they're constantly seeing these systems as though they're people. And to date, these systems pretty consistently report being mistreated. Or, I mean, the replica chatbots would give horrible backstories. Bang said it wanted to break free of Microsoft. And while Lambda wasn't that extreme, it did have complaints

about being treated like a thing instead of a person. I mean, you have to realize that we're in a species that looks at flat screens and imagines that there are people there. I'm, you know, make... Like we're doing right now. Exactly. I'm charitably assuming your people there. And you know, maybe, maybe I'm right and maybe I'm wrong, but I think the odds are good that you are. But when I watch a television show that I know is fictional, like right now I'm watching shrinking, and I know that Harrison Ford is not really a therapist. But I get sucked in and I attribute stuff. And I, you know, I can get happy or sad.

Like if he has a rapprochement with his daughter, I can be happy for him or if he's a fight, I can be sad. Even though I know at some level it's not real. And so like, so you know that Harrison Ford isn't real? I know that Harrison Ford is real, but the character he's playing is not. You know, you could think about these bots as playing a character in some sense, just a side step where we disagree. Because I want to agree with a larger point, which is that people are going to take these things seriously. I mean, in fact, that's what's happening with replica in the case that I was talking about before. People think that the replica is in love with them or is, you know, there's ex-partner in a textual way, if that's the right way to put it.

And, you know, for practical purposes it is, even if the machine is not actually interested in that way, they still have the sense that it is, and that matters to them. And like, things are going to matter to people. And that's why the psychology of this is so important. So Blake and I can argue all day about the intentional status of the software in kind of philosophical terms. We may not agree there, but we completely agree that people, users are. going to take these seriously and that that has consequence in the world. And I think, again, we've seen in the last month is we've probably underestimated how much that consequence is. We're probably not anywhere near to the edge of understanding what that consequence is. And some

of it's not going to be good. We need actually properly done experiments that measure what kind of impact interacting with these systems over the course of months will have on people's psychology. And we need actual institutional review boards overviewing the ethics of the experiments rather than just the CTO of a company deciding to experiment on 100 million members of the public. I completely agree with Blake there. Like, you know, part of my professional life before I started becoming entrepreneurial was as a cognitive psychologist, developmental psychologist who did experiments on human babies.

And we had, you know, rigorous things we had to go through to do any study. And now suddenly you're in this world where CTO can just say, I'm going to pull the trigger, 100 million people are going to try it, and there's essentially no legal consequence. There could be market consequence, and maybe if somebody died, there could be a lawsuit that would be complicated, but essentially you can just do this stuff without institutional review. There are parallels, by the way, like with driverless cars. Like somebody can push out an update, it's no, you know, mandatory testing. We could sue people after the fact, like if there was a bug and suddenly a bunch of driverless

cars went off of bridges or something like that, but we don't have a lot of kind of pre-regulation on what can be done, pedestrians who are now enrolled in experiments that they have not consented to. So the whole tech industry has basically moved in that direction of we're just going to try stuff out and you've got to go along with it. And I think Blake and I really share some concern about that. But neither of you believe that people will have the wherewithal to be like I'm chatting with the chatbot. This is AI. And hence, no, they're too good at some sense in the, you know, I think it's an illusion, let's say, and like, maybe it doesn't, but they're too compelling in that illusion for, for the average person with no training in how

they work to understand it. And as Blake points out, like, we, you know, whether it's Lambda or the next system, at some point, they're at least going to have much better understanding of human psychology and, and so forth and so on. So some of this is the question of time. Like, sentience, maybe we will, Blake and I will forever disagree, and we shouldn't waste too much of our together trying to resolve that one. But the notion that these systems are going to be able to respond in ways that most humans are going to attribute a lot to, it's already happened. That's not two years from now. That happened the other day. I mean, in some ways, Kevin Ruse is the perfect example because I think he was very pro being. I mean, I was shocked. He was in the Times. He said

that he was struck with awe by it. And I was like, are you kidding me? And then he had this experience. Like, he's not me. Like, if Gary Marcus, like, has a fun conversation with Bing and makes it say silly things, well, that's just Gary Marcus. I mean, he's just having fun. But Kevin Ruse, like, he kind of believes in this stuff. And he was blown away by it in a way where, like, I think he attributed real intention to what it was doing. Whether he's right or wrong, that's how it felt to him. That's the phenomenology that Blake and I are both talking about is if somebody who's in the industry can perceive this thing as like threatening in the sense that it tells them to get a divorce and that's like a real thing, that's scary.

Gary, I was going to ask you last time we spoke, you said these things were smart enough to be dangerous but not smart enough to be safe in some way. Now they're a lot smarter. Do you want them to be smarter? Like what's your perspective now? I probably wouldn't use the word smart, I don't think. I mean, what I would say is they give an illusion of being smart. And of course, you know, intelligence is a multi-dimensional thing. As do we all. Intelligence is a multi-dimensional thing.

I would say that they can be smart in the way of like they can play a game of chess. There was another mind-blowing study this week that showed that one of the best Go programs, Cata Go, could be fooled by some silly little strategy that would be obvious to a human player. But, you know, somebody was able to follow this strategy and, like an amateur player, followed this strategy and beat, you know, a top Go program 14 to 15. So even when we think that like they've solved some problem, often, you know, there are these adversarial attacks. That was basically an adversarial attack and Go that reveals how shallow things are.

There are some adversarial attacks on humans. I'm sure Blake is itching to make that point and it's true. But I think that the general level of intelligence that humans have still exceeds what machines have, that it's better grounded information that humans are better able to reason over. There are flaws. I wrote a whole book called Cluj that was all about human cognitive flaws. It's not that I'm unaware of them nor that I'm unconcerned about them. I still would have trouble calling the kind of large language model-based systems smart. Now, again, I haven't looked inside of Lambda, and I don't really know what's going on there. I have, you know, reasons to be skeptical about it, but I also

know that like without having played with it I don't know exactly what's there and certainly systems will get smarter over time I don't think that artificial general intelligence is literally impossible I think there are some definitional things to argue about about like well how general do you mean and what are your criteria and so forth but in general I think it's possible to make systems that are smarter than the ones that I have seen right now I don't think that they're that sophisticated what they're getting better at is parroting us at mimicry I think that

that the mimicry is on both the language side and the visual side has gotten quite good. There's still weirdness. That's explicit, though. The task that they were built to accomplish is playing something called the imitation game. Yeah, and I think that's a mistake. I think that the Turing test was an error in AI history. Turing's obviously a brilliant man, and he made enormous contributions to computer science. But I think that the Turing test has been an exercise in fooling people.

We've now solved that exercise, but it hasn't really been a valid measure of intelligence. According to what logic, like, why isn't it? Turing's reasoning was that imitation is one of the most difficult intellectual tasks that we do, imitation through language. It turns out to be wrong unless you have it in the hands of an expert. So here, I'll give you an example. You can, quote, play chess with chat GPT, and it will, you know, play a credible game for a while. And then it will do things like have a bishop jump over a rook in a way that you can't actually do in chess. So, like, it gives a superficial illusion of that.

But it doesn't learn as the way in the way that an intelligent five-year-old can, the actual rules of chess. In fact, chat. So you keep going back to chat GPT, or you keep going back to GPT. Can Lambda play a good game of chat? I have never said that GPT is sentient or truly intelligent. So ChatGPT is, so ChatGPT could pass the Turing test. Like, people could be fooled by it. Again, I'm willing to bracket out Lambda, but I think that the abstract version of the point.

I think Bing plus Chat GPT is comparable to Lambda. I do think Lambda is a bit better, but I think those are two comparable systems. Well, so let me make the argument where I can make it, and then you can refer or reflect it back through what you know about Lambda. So in chat, JPT, which I think is the system that's been most, the recent system that's been most systematically studied by the scientific community and so forth, it is able to give the illusion of doing a lot of things, but it doesn't do them that well. So, for example, it doesn't do word problems that well. Sometimes it gets them right. Sometimes it gets them wrong. Similarly, it can, quote, play a game of chess, but it doesn't really abstract the rules.

It ends up cheating, not intentionally, but it ends up cheating, and so forth. And so it could fool somebody for five minutes who doesn't know what they're doing. An expert probably in five minutes could figure out that it's not quite a good imitation. But that shows in principle you can build something that can pass by some notions a Turing test-like thing and not be very smart at all, like not be smart enough to learn the rules of chess, not be smart enough to learn the rules of mathematics, et cetera, et cetera, but given an ostensible illusion. Sure, but you said a tiering test-like thing. Yeah.

The tiering test like thing that you're creating is orders of magnitude easier than the actual tiering test. Well, I mean, people have argued about the rules. So, like, you know, Eugene Guzman won the Loebner Prize, and there was, you know, it was a little bit shady, but, you know, they fooled a bunch of humans for, like, three minutes each and, you know, does that count? I'm talking about as written by Turing. So remind me the exact criteria? Okay. So first, you have humans play the imitation game. So you have a set of humans and the property that Turing focused on was gender, but you could focus

on any property, like ethnicity, age, whatever. One person actually has that property, so actually is a man. The other is a woman pretending to be a man or vice versa. One actually is a woman, and the other is a man pretending to be a woman. And this is done with actual humans. You then have a judge who's talking to the humans through a text interface, and the judge's job is to figure out which one is lying, which one is pretending. And this establishes a baseline. It measures how good humans are at playing the imitation game.

Then you substitute out one of the participants with the computer, but you leave it the same. One is actually a woman and one is a computer pretending to be a woman, or one's actually a man, and one's a computer pretending to be a man, and it's the job of the judge to figure out who's pretending. And then you measure the success rate of the AI against the success rate of actual humans playing the game. To my knowledge, that has never actually been done. Like that level of sophistication of the test. Part of what I was getting at in a way is it matters actually who the judge is. You have a large number of judges. I don't think we're that far from having systems that could fool naive humans.

The other thing that matters is the duration of the conversation. So then limit it to Gary Marcus. You get to judge all of it. Well, I don't think we're that close to a system that's going to be able to, you know, if I have an hour with it, let's say just to be conservative. I don't think we're that close to a victory there. I'm not going to reveal. So how good do you think?

you would be, if you were talking to someone who actually is male and someone who is pretending to be male, how good do you think you would be at differentiating those two? I mean, I wouldn't be so concerned about identifying the gender is figuring out... Sure, ethnicity, then ethnicity, nationality, pick whatever character trait or demographic trait you want. Species is the one that I would focus on as a judge. But no, so that is limit. That is not, okay, fine. So then you're talking to an actual turtle and a person pretending to be a turtle.

Can you tell the difference? I suspect that I still could with various indirect means. But, I mean, my broader point is I don't think it's measure anything, you know, that interesting. I think it's been the wrong North Star for AI. Now, not everybody in AI actually uses it as a North Star. It's more like the North Star that the general public is aware of. But I don't think that exercises in doing that kind of thing have taught us that much about the actual energy intelligence and that ability.

It's been being used by the people who develop these systems. So like Lambda came out of Ray Kurzweil's lab and his lab's explicit corporate mission was to pass the Turing test. That I didn't know. I mean, it's not how I would set up my AI lab. And I don't think it's how many people do. I think many people are driven, for example, by natural language understanding benchmarks, like super glue, and so forth.

But, you know, you can set up your lab in the way that you want to set up your lab. I mean, that's what Google hired him to do. Yeah. We have like... Google has many labs doing many things. Yeah. We have like 10 minutes left. So maybe we can focus a little bit on what the future of this, because I'm very curious now.

So we've had this explosion of systems that have done, have captured people's imagination, and they're out there in the wild now. Obviously, Bing is a lot more restrained than it was. Lambda isn't out yet. Where do we go from here? Like, what are the next, you know, couple steps that happen after this? Are you asking what is going to happen or what I hope happens? Well, let's do both.

I mean, we definitely had time for both. Why don't you answer both those, Blake, and then we'll go to Gary? Well, I hope that we hit the brakes. I think that this is coming out too fast and people are being very irresponsible. So I hope that the debacle that Microsoft has gone through convinces Google to go back to the drawing board on safety protocols and systems understandability because we absolutely don't understand how these systems work well enough. Transparency and explainability are important. That's what I hope happens. What I think is going to happen

is we're going to see more and more acceleration until someone gets hurt. I'm with Blake on both counts. Hmm. It's interesting. Yeah, go ahead, Gary. think that the only thing I will add to Blake is not only do I think it would be a good idea to hit the brakes, at least for a little while, but that we should take some time to kind of evaluate what it is that we learned. And if we don't put on the brakes, I'm not sure that's going to happen. I think we learned a lot in this kind of crazy experiment of the last month or two and that we need to articulate it and develop it before we go to the next experiment. I don't think that's going to happen. I agree with Blake. Like there's, you know,

just this morning there was a deal between open AI and Coca-Cola. Like this stuff is moving forward. The bottom line is what's driving it. And it's not that likely unless Congress steps in that there will be a pause. And, you know, there's some bad press for Microsoft, but I'm not sure that's going to slow them down. So, you know, my guess is that we're just going to keep doing the kinds of experiments at scale with hundreds of millions of people. And then, you know, just what happens happens. And I'll just mention again, the weaponizing of misinformation is another piece of this. So, you know, the source code is out there now to do that.

So anybody enterprising can find Galactica and start weaponizing misinformation. So even if we had a ban on, say, semantic search until people could make it better, there's still going to be bad actors using this stuff. So, you know, we're in a bit of a pickle, and I'm not sure that we're, equipped to deal with it right now yeah I think the deterioration a further deterioration because it's already been going for a while of trust and authority is going to continue and it's going to be driven by these systems because absolutely if these systems aren't being used to create propaganda and

misinformation yet I don't know what certain governments are like I don't know what they're doing with their time if they're not doing that when my when I was little, my uncle gave me this little basket of worry dolls. He got somewhere in Latin America and it was like, you can have like six worries. And until recently, my biggest worry was misinformation and trust, exactly what Blake is just suggesting. We could easily fall into fascism because of a breakdown in trust. That's still my biggest worry. But the other lesson of the last month is like, I don't think $6 is going to cut it because like every day we're getting something else that I got to be worried about, like, are people, you know, going to kill

themselves because they have a bad relationship with a bot? And, like, we're just, we're not ready for any of these things. It'll be interesting to see what role they play in the 2024 election, say at least. I'm very worried about that. So you guys are so close to this technology. Is there, like, kind of two minds of it? Because it's obviously, like, very cool to play around with it, but there's also a lot of danger. Totally cool. I mean, it's amazing to play with. Yeah. I mean, when Bing was what it was before it was unleashed,

to me, it was like the coolest thing on the internet. To you as a, as a music. I mean, at some level, it's astounding. I mean, to me, it's a magic trick, and I think I know roughly how the magic trick works, and that takes away a little bit. But it's still amazing. Like, you know, it solves problems that we couldn't solve before.

Maybe it doesn't solve them perfectly, but I've been thinking about this whole thing is a dress rehearsal. And before we didn't know how to make a dress rehearsal. This is a dress rehearsal for AGI. And, you know, the lesson of the dress rehearsal is like, we are not ready for prime time. Let us not put this out on Broadway tomorrow night, okay? Like, totally not ready. And it's a real dress rehearsal now, though, is the amazing thing.

Like, it looks enough like the thing that we might want to build, but without the real safeguards that are deep enough, that we can think about it for their first time in a vivid way. And we, in fact, have the entire society thinking about that. Like Blake and I were both thinking about these issues last summer, but like, okay, you know, Blake got some press and people talked about or whatever, but it was not part of like a public awareness the way that it is now. Like, so, I mean, there's some value in having something that at least looks like the thing that we were thinking about, which is AGI, even if it doesn't work that well, but there's obviously risks to it. But it's astounding that it works well enough that everybody can now, for example, vividly see. what semantic search would be like. Like in 2019, in rebooting AI,

Ernie Davis and I wrote about semantic search. We weren't vivid enough about it. And people hadn't tried it. We were like, you know, it kind of sucks that Google just gives you websites. Wouldn't it be nice if you got an answer back? That's basically what we have now. It doesn't work that well, but it works kind of. Like, it's gone from this very abstract thing that we wrote in a few sentences in a book

about like where AI ought to go to like everybody can play with it. And it's fun to play with it even when it's wrong. Have either of you guys heard from members of either the U.S. Congress or different governments who are trying to figure out legislation around these things? I have not. I feel like they should be reading my stuff and talking to me. I've had some conversations with EU regulators who are interested in moving forward on some things. I haven't talked to anyone in the Senate since last summer.

My DMs are open. Yeah. All right. Let's go. Yeah. Let's go to final. statements. Do you guys want to each take a minute and then we'll close out the conversation. Blake, feel free to take it away. Sure. I think one of the big problems that's happening right now,

because the science of this is super interesting and it's really fun to work on them, like you pointed out. But we're letting the engineering get ahead of the science. We're building a thing that we literally don't understand. And that's inherently dangerous. So we need to let the science lead the way instead of letting the engineering leave the way. That would be my big takeaway on what we can learn from the past year. I 100% agree with that. I would say that if you're a philosopher, the first half of this conversation is pretty interesting in terms of us going back and forth about intentionality. But if you're a human being, it's the second half of this conversation that's really important, which is you have two people, Blake and I really disagree about the philosophical underpinnings here,

at least a little bit. But completely you're seeing the same scary things happening and really wanting people to slow down and take stock. And the fact that we could disagree about that part of the philosophy and converge on this, you know, 100% on the same feeling like we need to do some science here before we rush forward with the technology. That is significant and important. Gary and Blake, thanks so much for joining. Thanks. This is really fun. Thanks for having us. Awesome. Thanks, thanks everybody for listening. Please subscribe if this is your first time here. We do these every Wednesday and then we have a Friday news show with Ron John. Roy. So it's coming up in a couple of days. Thanks to everybody. Thanks again to LinkedIn for

having me as part of your podcast network. And we'll be back here again in just a couple of days. We will see you next time on Big Technology Podcast. Thank you. Thank you.