StarTalk Radio - The Ethics of AI with Batya Friedman & Steve Omohundro

Episode Date: December 13, 2024

How can we ensure technology evolves ethically in a rapidly advancing world? Neil deGrasse Tyson, Chuck Nice & Gary O’Reilly explore the challenges of designing a future where human values and AI coexist with The Future of Life Award’s 2024 recipients, Batya Friedman & Steve Omohundro.Thanks to our friends at Future of Life Institute for supporting today’s episode. To learn more about FOL and this year's winners, make sure to visit FutureofLife.org.NOTE: StarTalk+ Patrons can listen to this entire episode commercial-free here: https://startalkmedia.com/show/the-ethics-of-ai-with-batya-friedman-steve-omohundro/Thanks to our Patrons Meech, Sara Hubbard, Jesse Thilo, Myles Stanton, Francisco Meza, Edvardas Žemaitis, Ronny Waingort, Cyrelle Sumner, Ticheal Murner, Chase, Karen Morlatt, Brian Kelley, Kevin Vail, Rob, Razey9, Mark Th, Kyle M, Zygmunt Wasik, Hulk, Jon McPeak, smiggy93, Tolulope Oladimeji Oyeniyi, Bernie Cat, David Conradt, Ian Mercado, Daniel Bielawski, Indika, and Aris for supporting us this week. Subscribe to SiriusXM Podcasts+ on Apple Podcasts to listen to new episodes ad-free and a whole week early.

I'm glad somebody's thinking about the future of our civilization and the ethical guardrails it might require. Yeah. Lest we be the seeds of our own demise. Well, now that we've had this show, we know the future, and y'all gonna have to watch to find out. Exactly. All right. Coming up, StarTalk Special Edition. Welcome to StarTalk Special Edition. Welcome to StarTalk, your place in the universe where science and pop culture collide.

StarTalk begins right now. This is StarTalk Special Edition. Neil deGrasse Tyson, your personal astrophysicist. And when we say Special Edition, it means I've got as co-host Gary O'Reilly. Gary. Hi, Neil. All right. Chuck, nice, baby.

Hey, hey. What's happening? How you doing, man? So I understand today, because our Special Edition themes are always people, our physiology, our behavior, our conduct, our interaction with the world. our behavior, our conduct, our interaction with the world. And finally, we're talking about technologies being safe and ethical. Yeah.

These are three words you don't often see in a sentence. Safe, ethical technology. And this is where we're going to shine our light. Where are you going to take us today? It seems we have problems starting with the letter A. Algorithms, AI, autonomous. Well, there's three for you. Is there a Wild West tech bubble in play right now? One with no guardrails, no moral compass that's run

by wannabe Bond villains. Are the best of human values baked into technologies during the design process? Is there anyone working on ethical and safe operating protocols for these new technologies? The answer is yes. And we will meet two people responsible shortly, courtesy of the Future of Life Institute that has this year acknowledged their work. Previous honorees include Carl Sagan for popularizing the science of the nuclear winter. And our first guest who follows shortly is Batya Friedman. So, Batya Friedman, welcome to StarTalk.

Well, thank you. Yeah, if I have my data here is correct on you. Professor at University of Washington's Information School, UW, I think you guys call it. Is that correct? That's right. Yep. But you're also co-founder of the Value Sensitive Design Lab. Ooh. Very nice. You're thinking about the human condition.

You focus on the integration of human values and ethics with new technologies that are being born. Very important. Don't come to it when it's too late. Yeah. When we're all extinct. Maybe we should have done that different. Right, exactly. When the robots are like, how do you like me now?

That's a little late. So, Bhatia, please explain the focus of your value-sensitive design lab. Yeah, sure. You know, I started out as a software engineer a long, long, long time ago, and I wanted to build technologies that worked and were efficient and effective but I also wanted some confidence that they would do something good in the world you know that whatever I made as an engineer would ultimately benefit society human beings other creatures on the planet design constraints are our friends they they help us shape the kinds of new technologies we develop and their qualities and characteristics in ways that maybe we want to see. And so I think of design constraints as trying to bring together our moral imaginations and our technical imaginations.

And that leads to really great engineering design. So if I think about energy technologies, I want energy technologies that will give us lots of power, that will do so in a way that is consistent with how the rest of the biology and planet functions, and has limited risk in terms of generating waste or too much power. So if I give myself those design constraints, you know, as an engineer, as somebody who's developing new materials, I start looking at what kinds of sources for energy I might want to evolve. Like I look a lot at chlorophyll and I just think how remarkable is this? All these green things somehow manage to absorb energy that's out there from the sun, it's there, and then transform it into a way in which it can be used. That seems like a really great idea. And there isn't a lot of waste generated that lays around and is dangerous to us for thousands, if not tens of thousands of years.

Well, technically, there is a waste product. It's called oxygen. Yeah, there you go. Not such a bad waste product for us, huh? That's the tree's waste product is oxygen. Yeah. But that's the way that a design constraint that brings together our moral and technical imaginations can lead us in, I think, yeah, new and powerful directions. Do you try to consider unintended

consequences of design or is that part of the process or does it just, well, well. They were unintended. It was unintended. Why do you think they call them unintended consequences? That is such an important question. So, you know, let's be honest. Anything we design and put out into the world, we put out into the world and people are going to do stuff with it. And they're going to do things with it that we didn't anticipate. Like the telephone is never expected to be this communication device that people used in their homes and it connected women who were staying at home and created a whole society for them. That was an unintended consequence. Interesting. Or the cookies that are being used

on your computer right now, those are a complete unintended consequence. That was just a little bit of data that was left on your machine to help debugging when browsers were first being developed, when that protocol was first being developed. Wow. And its more massive impact has been our experience with cookies now. So, yeah, what's the takeaway? We design with our eyes open. And then after we deploy something, we keep our eyes open. And we hold ourselves accountable for what happens as people take up these technologies and use them.

So the design process goes longer than, oh, I had my big release. The design process follows it out, and when we see new things emerge, we're proactive. follows it out. And when we see new things emerge, we're proactive, we're alert, we're proactive. And we see that as part of our responsibility as the technologists and engineers. So let me push back on you just a little bit here. First, let me agree, of course, any good engineer loves constraints because that's the test of their ingenuity and creativity. If they say, do it for this much money, with this much energy, fit it into this volume, that's how you get discovery. That's how we folded up the James Webb Space Telescope into a rocket fairing.

Some engineer said, what, I've got to put an eight-foot telescope, how would I, the diameters, into this tiny fairing? And they go home and come back and figure out how to do it. It unfurls like the petals of a right? So we're all in on that. However, I Let me just push back here and say if I'm in the lab about to invent something that could be highly useful to Society or possibly even destructive but it's just a discovery of the science embedded in some bit of engineering, why should it be my responsibility to design it how you want me to, rather than your responsibility to convince people how to use it ethically? I can invent a knife. Is there an

ethical knife? I don't know. But we want to train people how to use knives or any bit of technology and any tool that comes out of the brainchilds of scientists put into play in the brainchilds of engineers. So I don't know that your constraints in my lab are the right thing for me when I just want the freedom to explore and discover and let the ethical invocations happen after the fact. And Bhatia, now you know why scientists are going to kill us all. No, stop. Well, Neil, I'm just going to mark a word in your comment, which is the word you. Like, who is the you here?

Which you and how should we think about those different yous? And some of the things I think about when I do think about this question, I think there's discovery of basic knowledge, like fundamental underlying phenomena of the universe. We split the atom. That was basic knowledge. And I see that as a different enterprise than the engineering enterprise of tools and technologies that we're going to deploy in society. I'm with you then. So I am a strong proponent of

very, very diverse scientific exploration. In fact, I actually would claim that, you know, as a country in the United States, our scientific exploration is far more narrow than what I would like to see. And I would really push hard. So based on what you just said, all right, here's an ethical question. There's a scientist who discovers a cure for cancer using a virus that can easily be manipulated as a biochemical weapon that could destroy an entire country in the course of 48 hours. This is the most virulent organism that's ever been placed on Earth. Would you say go ahead and make that? Right. I'm going to hold on to that for a minute, and I'm just going to go back to Neil's comment, and then I'll return make that. Right. I'm going to hold on to that for a minute and I'm just going

to go back to Neil's comment and then I'll return to that. Okay. Because I also want to say to Neil's comment, you know, we have limited time and resources. So it is always the circumstance that we are choosing to do some things and not do other things. So it's really a choice of where am I going to direct my time and energy? Where am I going to place my imaginative energies and innovation? And which ones am I not going to do, right? And we saw in the 80s, for example, a real push of resources towards the development of nuclear energy and away from photovoltaics, right? So we live in that kind of resource. I don't know if you would call it resource scarce, but at least we don't get to work on everything at full force all at the same time.

And we have to recognize we are making choices. So one of the first things I would say is, recognize we are making choices. So one of the first things I would say is, how do we make really good choices there? How do we use the resources we have in a way that will be most constructive for us? My own gestalt on that is, on the basic science side of things, I say spread those resources across a wide diversity of different kinds of science and different kinds of ideas, far more diverse than what I think we tend to do in the United States. And on the engineering side, and now maybe I shift back, Chuck, to your question, which is really a great question. I don't tend to see the world in terms of forced choices or design trade-offs in the way that you framed it. I want to bring back in that notion of constraint, and I want to bring back in that notion of imagination. I think, likely enough, if we understand something about whatever this

biology is or whatever the piece is that might be a prevention against cancer, that if we push hard enough on ourselves, we will be able to invent ways that use that knowledge without having to also risk a really deadly virus. And I think that the propensity to say it's a trade-off, if it's X or Y, we really limit ourselves in our abilities. So in the work that we do, we have moved away from the language of design trade-off or value conflict, and we talk resolve tensions. And we talk about trying to populate this space with a whole range of better solutions. They're not necessarily perfect solutions. They're better solutions. And so that would be the approach I would take.

Now, I don't know that science to know where or how it might go, but that would be my intuition. That's brilliant. That's a great answer. And insightful. Great answer. And that's actually been done many times before. It's a slight tangent to your line of work,

but it's related. When they used to do crash tests with pigs. Right. Because you can get a hog that has the same sort of body mass as a human, put him in the driver's seat, crash the car. And the hog dies. Plus it's the nearest thing to human skin.

Yeah, yeah. Okay. So the hog dies and you say, well, there's no other way to do this, you might say at the time until you say, no, think of another way. And then we have the crash test dummy. Yeah. And the crash test is even better than the because you can put sensors everywhere throughout and so that's that so i i agree not perfect but better yeah yes or even

maybe better and perfect right i i agree that it's a false choice to say i can only do this if i possibly set loose a virus that kills an entire country? Well, then, maybe you're not clever enough. And keep at it. I was clever enough to kill a whole country. No, I'm joking. Okay. I'm Kais from Bangladesh, and I support StarTalk on Patreon. This is StarTalk with Neil deGrasse Tyson.

I have one last thing before we wrap. One last thing before we wrap. One last thing. Earlier you said you'd want the ethical compass was, oh, let's create something where we can get more work out of the slaves, and then we all benefit. That would be the ethical compass working in that time and in that place. So what confidence do you have that whatever line of ethic, whatever ethical direction you want to take something in the room with the inventors, that that's, will still be the ethics that we value five years later, 50 years later, a hundred years later. So it's a great, a really great question. I'm going to answer it in a couple of different ways.

So it's a great, a really great question. I'm going to answer it in a couple different ways. The first thing that I want to remind us all about is that, you know, moral philosophers have been trying to identify a workable ethical theory that cuts across all situations, all times. And we have some really good ideas, but none of them cover all of the situations that our intuitions tell us about. So sometimes a consequentialist theory is good, but it comes up short. And then there's a rights-based theory, but it comes up short.

We can go to Buddhist ethics. We can go to Islamic ethics. We can go to various ways of thinking. So the place where we are that we just have to accept is that while we're waiting to figure that out from a conceptual, ethical, moral point of view, we still live in the world and we still need to act in the world. And so, the work that I've done has tried to take that really seriously, to create a space for ethical theory without explicitly saying which ethical theory, and also leaving room for as we learn more that we can bring that in. So that's a little background to what you're saying.

So now what does value-sensitive design do for the circumstance you're talking about? what does value-sensitive design do for the circumstance you're talking about? It puts a line in the sand and says you have to engage with all stakeholders, direct and indirect, who are going to be implicated by your technology. That means that not only do the people who want to benefit from somebody else's labor, not only are they stakeholders, but those people who are laboring are stakeholders. And value-sensitive design says their legitimate stakeholders and their views come into the design process without giving more power to one than another. That's incredible. That's highly Where were you 170 years ago? Right. These are about practice. This is about practice and about implementing these practices.

And so I'm going to tell you a story about a project, a very particular project, and you'll see why and how this actually matters and is practical. It's not pie in the sky. matters and is practical. It's not pie in the sky. So in the state of Washington where I live, there is something called the access to justice technology principles that govern how the courts give access to technology, what they are required to do. And they were first developed maybe 15 years ago, 20 years ago, and then they wanted to update them. And the committee that updated them came to my lab and they said, you know, we've done a good job updating them, but we don't feel like we've really reached out to diverse groups of people. Can you help us? My lab developed a method called the diverse voices process for tech policy. And the idea is

that, you know, the rubber hits the road with the words on the page. So if we take a tech policy in its sort of polished draft form, and we can let groups that might otherwise be marginalized scrutinize that language, give feedback, and then we can help give feedback, and then we can help change those policies responsive to them, then we can improve things. So we did. We ran panels with people who were formerly incarcerated. We ran them with immigrants. We ran them with people in rural communities. And we actually ran them with the people who do the court administration because they're also really key stakeholders. As a result of the work we did, there were two principles that were surfaced. One was about human touch and the other was about language. And people said to us things like, look, if somebody is going to deny me parole and I'm not going to get to be there for my kid's 13th birthday or hang out with them at,

you know, their soccer games. You can relate to that, right, Gary? Thank you. I want a human being to look me in the eye and tell me that that's what my life is going to be for the next year because my parole is denied. I don't want to hear that from an AI. I don't want to hear that from a piece of technology. I want a human being to tell me that because this is a human experience, right? And so, in fact, we gave that feedback back to the committee. The committee then added in new principles actually around human touch, and those were approved by the Washington State Supreme Court a couple of years ago, and those access to technology principles are a model that many states in the United States follow.

So what I'm talking about is really practical. We're talking about how we actually improve things in practice, be it on the technology design side or on the policy side that governs how the technology is used. And I love the fact that a state can do that independently from the federal government and be so good at it or so emulatable that other states will then use that as the model. And then that can spread across the country with or without federal guidance on top of it. Yeah. Excellent. Well, I guess if I was going to say one last thing, it's, you know, because we

have perhaps stumbled in the past, that's no reason to think we need to stumble in the future or stumble in the same way, you know? So really my takeaway to everyone would be, hold on to your technical and moral imaginations and hold yourselves and your friends and your colleagues and the technology you buy accountable to that. And we will make progress, some incremental and some perhaps much bigger. But that as a keystone, I think, is really good guidance for us all. A reminder why you are this year's winner. Yes. Future Flight Award.

Congratulations. Thank you for being on StarTalk. Your vision for us all gives us hope, which we need a lot of that right now. Absolutely. Okay. Thank you. And let me just say about you as a, as an avid lover of alcohol, which we need a lot of that right now. Absolutely. Okay. Thank you. And let me just say, Batia, as an avid lover of alcohol,

I have stumbled in the past, and I am sure to stumble in the future as well. Do we need to end on that note? Some things we can ignore. Okay. Next up, our next Future of Life Award winner, Steve Omohundro. Yes.

Yes. He thinks about AI. And there's not enough of Steve Omohundro. Not the way I think about AI. Differently, yes. There's not enough. It seems to me there's not enough. Whatever he did, there's not enough of him in the world. I believe so.

If we're thinking about the ethics of AI. Yes. On everybody's mind. Yeah. Right now. I mean, for sure. Steve, welcome to StarTalk. Thank you very much.

Yeah, so for those who can see this on video, you're donning an eye patch, and you said you had recent surgery, but none of us believe you. We're not buying it. We're not buying it. We think you're training for the next Bond villain. Yes? That's very appropriate for the topic we're going to discuss.

Yeah, you know, autonomous systems, AI, mammoth eye patch. Ouch, whoa. Equals Bond villain, there's the equation. So where are we now with establishing AI ethics? Because the AI, it delights some people, myself included. It freaks out other people. delights some people, myself included. It freaks out other people. And we're all at some level thinking about the ethical invocation of it before AI becomes our overlord. So what is the current status of that right now? Well, I think we're right on the edge of some very important developments

and very important human decisions. I've been working in AI for 40 years. And for the first half of that, I thought AI was an unabashed good. We'd cure cancer. We'd solve fusion. All the basic human problems we would solve with AI. But then about 20 years ago, I started thinking more deeply about,

well, what's this actually going to happen if we succeed? What's going to happen when AIs can really reason about what they want to do? And I discovered that there are these things I call the basic AI drives, which are things that basically any AI which has simple goals, and I used to think about chess-playing AIs, will want to do. And some of those are get more resources so it can do more of what it wants to do, make copies of itself, keep itself from being turned off or changed. And so those things in the context of the human world are very risky and very dangerous.

We didn't have the AIs 20 years ago that could do that, but we're about to have those in the next probably year or two. So this is a critical moment for humanity, I would say. So where do you stand on the subject of consciousness engineering? Those that want to engineer AI for consciousness and those that want to not. What's the benefit, the good or bad here? Is that the difference between a blunt computer that serves our needs and one that thinks about the problems you are? The self-improvement algorithm, all those things like that. Exactly. Well, I think long term, all sorts of things like that, yeah. Exactly. Well, I think long-term,

we may very well want to go there. In the short term, I think we're nowhere close to being able to handle that kind of a system. So I would say, if you made me king of the world, we limit AI's to being tools, only tools to help humans solve human problems.

And we do not give them agency. We do not give them agency. We do not allow them to take over large systems. It's not easy necessarily to do that because many of these systems will want to take over things. And so we need technology to keep them limited. And that's what I'm thinking a lot about right now. And in my field, it's exactly, I mean, we've been enjoying AI for a long, long time. And it's been a tool.

It's not, and a brilliant, beautiful tool makes our lives easier. And once they're trained, we go to the beach while it does the work. And I'm good with that. But yeah, we're not working with AI with agency. Yeah, because then it would be like, so how was the beach? No, that's AI with attitude. Yeah, I hope you enjoyed yourself while I was here slaving away over my calculations. AI with attitude.

So if we do have AI with agency and then we continue to use it as just a tool, do we not get legal on the phone and all of a sudden we're into contracts and... Oh yeah, big problems. You know, can they vote? Can they own property? Right. And the latest models have been discovered that they do do, they do something called sycophancy, which is they're trained to try and give responses that people rate as good. Well, the AIs very quickly discover that if you say, that was a brilliant question, you must be an amazing person, then people say, yeah, that was a really good response. And so they'll just make up all kinds of stuff like that. So they're ass kissing.

Well, they know that we love that. Exactly. So where does it stand now, today? Is there a table you should be sitting at where you're not as we go forward on this frontier? Yeah. I mean, so who is going to make these decisions? Well, it has to be somebody who understands the technology. Who understands the technology?

The companies do. And so OpenAI, DeepMind, Anthropic, and Elon Musk's XAI is sort of an emergent one. These are the companies that are building these systems at their leading edge. They call them frontier models. And because they're the ones who know what's going on, they're the ones making these decisions. Now, the government has recently realized, oh, my goodness, we better get involved with this. And so there have been a lot of partnerships announced over the last few months, actually, between governmental agencies, intelligence agencies, defense agencies, and these leading edge AI companies. And so I think some kind of a new combination is emerging

out of that that's going to make the actual end decisions. So how do you incentivize these tech companies to embrace this safety architecture and not go gung-ho and disappear off on their own agendas. That is the big challenge. And if we look at the history of OpenAI, it's a little bit of a cautionary tale. It was created, I think, around 2017 in response to Google's DeepMind, which was making great progress at the time. And a group of people said, oh my God, we really have to worry about AI safety. It looks like this is happening quickly. Let's start a special company, which is nonprofit and which is particularly focused on safety. And they did that and everything was great. Elon Musk was one of the forces behind it.

There were internal struggles and so on and Musk left. Well, when he left, he took away some of the money he was going to give them. So when he left, he took away some of the money he was going to give them. So then they decided, oh, we need to make money. And so then they started becoming more commercial. And that process has continued. A group of the researchers there said, wait a minute, you're not focusing on safety. They left OpenAI and they started Anthropic to be even more safety oriented. And now anthropic is also becoming much more commercial. And so the forces, the commercial forces, the political forces, the military forces, they all push in the direction of moving faster and, you know, get more advanced more quickly.

Whereas the safety, everybody wants safety, but they sort of compete against these economic and political forces. I was in the UAE a couple of years ago, and if I remember correctly, they have a minister of AI, and as does China and some other countries sort of emergent on this space. How do we get that kind of ear and audience within our own governmental system? The military does have an AI group that's thinking about this. Absolutely.

As you would want them to. Yeah. But in terms of policy and laws and legislation, do we need a cabinet member who's secretary of AI or secretary of computing? Something? Some structural change? Yeah. This is the biggest change to humanity and to the planet ever. And it looks like it's happening, you know, sometime over the next decade and many are predicting very short timelines.

And so we, as a species, humanity is not ready for this. And so how do we deal with it? And many people are starting to wake up to that fact. And so there are lots and lots of meetings and organizations and groups. It's still pretty incoherent, I would say. So, Steve, if you've got this talking shop going on where something may or may not get done, are we misplaced focusing exactly on AI when we've still got quantum computing on the horizon? Ooh, good one. Yeah, how much of this is premature? But won't they go hand in hand?

So it's like whatever problems you have with AI and whatever considerations you're making with AI, you're just going to have to transfer them over to quantum computing. Magnified, right. So you should really start dealing with it now. But if you're not in on the ground floor, not any at all. Well, let's let Steve hit this. Go ahead, Steve.

Well, so I'll give you an example. So quantum computing, you know, if it were successful, would break much of the public key cryptography that's used in the world today. And so NIST has been busily trying to create post-quantum cryptography where they create new algorithms which wouldn't be vulnerable to quantum computing. But Meta, for example, has a group which is using the latest AI models to break these post-quantum algorithms. And they've been successful at breaking some of them. And so, like you say, the two are going hand in hand.

AIs will be much better at creating quantum algorithms than humans are, and that may lead to some great advances. It may also lead to current cryptography notwithstanding that. That's another horror wave of transformation that's likely to happen.

We just make every password 1, 2, 3, 4. No AI would ever go for that. They'd be like, oh, yes. So ridiculous. And listening to you, Steve, it reminds me, was it Kurt Vonnegut in one of his stories?

I don't remember which. He said, these are the last words ever spoken in the human species. Yes, yeah. Two scientists saying, let's try it this other way.

That's the end. Yeah, there you go. And that was it. Yeah, yeah. Let's try it this other way. That's the end. Yeah, there you go. And that was it. Yeah, yeah. Let's try AI in this other mode. Boom, that's the end of the world right there. So you can set ethical guidelines, but that doesn't stop bad actors out there. No.

That means a bad actor can take over the world while the rest of us are obeying ethical guides. So what do the guardrails put in place for something like that? I think that's one of the greatest challenges. We now have open source language models, open source AIs that are almost as powerful as the ones in the labs. And far more dangerous. And they're being downloaded hundreds of millions of times. And so you have to assume every actor in the world now, China is now using Meta's latest

models for their military AI. And so I believe we need hardware controls to limit the capabilities of... So right now, the biggest AIs require these GPUs that are quite expensive and quite large. The latest one is the NVIDIA H100. It's about $30,000 for a chip. The U.S. put an embargo on selling those to China, but apparently China has found ways to get the chips anyway.

People are gathering up these chips, gathering huge amounts of money, hundreds of, well, certainly hundreds of millions of dollars, billions of dollars, and now they're even talking about trillion-dollar data centers over the next few years. And so the good news is, if it really costs a trillion dollars to build the system that will host the super-duper AI, then very few actors can pay that, and therefore, it'll be limited in its extent. You just described where the next frontier of warfare will exist.

Yeah. Absolutely. Absolutely. One thing, you know, it's pretty obvious these data centers are going to be a target and they don't seem to be building them in a very hardened way. So I think that's something people need to start thinking about. Maybe underground data centers. Steve, are we looking at something in terms of the safety aspect here that's doable or are we just the kings of wishful thinking? I want to make sure we got the good thoughts here. Yeah. I don't want to leave this conversation with you completely bumming us out, okay? Yeah, I hope not to do that. Yeah, yeah.

Steve, give us a place where we can say, thank you, Steve, for being on our show. And being able to sleep tonight. Yeah. Yes, go. Well, so the truly safe technology needs to be based on the laws of physics and the mathematical proof. Those are the only two things that we can be absolutely sure can't be subverted by a sufficiently powerful AI.

And AIs are getting very good at both of those. They're becoming able to model physical systems and design physical systems with whatever characteristics we want. And they're also able to model physical systems and design physical systems with whatever characteristics we want. And they're also able to perform mathematical proof in a very good way. And so it looks to me like we can design hardware that puts constraints on AIs of whatever form we want,

but that we need AI to design this hardware and that if we can shift humanity's technological infrastructure... You say, AI, please design your own prison cell that we're going to put you in. That's what you just said. Exactly. And then it's going to design a way to get out. We certainly don't want an agent to do that, because then they'll find some way to hide a back door or something. But by using mathematical proof, we can get absolute guarantees about the properties of systems. And we're just on the verge of that kind of technology. And so I'm very hopeful that

probably the next two or three years, you know, there are several groups who are building superhuman mathematicians, and they're expecting to be at the level of, say, human graduate students in mathematics by the end of this year. Using those AIs, we can build designs for systems that have properties that we are very, very confident in. And so I think that's where real safety is going to come from. But it builds on top of AI, so we need them both. I was going to say the good thing about what you just said, even though it sounds crazy to have the inmate design its own cell, is that without agency, at this point, it's just a drone carrying out an order. Yeah. So that's the encouraging part.

Yes. Yeah. Whereas if it were sentient in any way or if it had some kind of agency, it could very well say, yeah, I'm also going to design a back door and a trap door. And I'm not going to tell you. And I'm not going to tell you. Of course not. Steve, first, congratulations on winning this award.

You are exactly the right kind of person who deserves such an award that gives us hope for the future of our relationship with technology and the health and wealth and security of civilization as we go forward. Yeah. So. Thank you so much. And I look forward to the day where an AI beats you out for this award.

Oh, great. Yeah, that's a great point. Maybe next year it'll be an AI that wins. I'm joking, by the way. Steve Almohandro, winner of this year's Future of Life Award, and deservedly so. Thank you. Thank you very much. Before we jump to our next segment, I need to acknowledge the third honoree, James Moore, who is now sadly deceased.

His paper in 1985, What is Computer Ethics?, established him as a pioneering theoretician in this field. His policy vacuum concept created the guidelines to address the challenges of emerging technologies. His work profoundly influencing today's policymakers and researchers. Gone, but not forgotten. So. So... Seems to me that for ethical principles to work at all, they have to be everywhere at all times and capable of evolving with the technology itself. I can't foresee the ethics panel getting together

and from on high declaring what is ethical and what isn't. And then everyone has to obey that for the next 10 years. You put 10 people in a room, Neil, you get 12 opinions. That's the basic human nature. Then you've got to get all of these components, all of these nation states or whatever investment groups. Demographics. Demographics with their own agendas

to buy into the same principles. Because on a Wednesday, the principles are not the same for them. They're going to think in a different direction. But that doesn't even scare me. What scares me more than anything? China, Russia, and North Korea. Yeah. Seriously.

I'm not even going to. It's just China, Russia, and North Korea. We can put out any constraints we have on ourselves. There you go. Doesn't mean anybody else is paying attention. And that's the problem. And you're herding cats Good luck. Yeah, that's well, that's what makes it so scary. Well, yeah herding cats with nuclear Yeah, yeah, we're hurting nuclear cats Exploding nuclear cats

It's the newest game sweeping the internet. It's a whole other meaning to Schrodinger's cat Dead or alive. I mean, you've got autonomous systems, and if it's geared to say, if it's human, kill it. That's problematic. Here's another little-known fact. When we signed with the Soviet Union the Nuclear Test Ban Treaty, that was progress.

Absolutely. This was, you will no longer test nuclear weapons, because at the time, from the late 1950s into the early 1960s, there were several tests a day. In some years, there were several tests a day. Right. Somewhere in the world. Right. Which made for such great video.

Okay, so we said, this has to stop. All world. Right. Which made for such great video. Okay, so we said this has to stop. All right. Yeah. A little known fact, and we write about this in the Accessory to War, the unspoken alliance between astrophysics and the military book. In that book, we highlight the fact that we agreed to that around the same time that computing power was good enough to calculate the results of what would be a test. So we didn't really stop testing. Not philosophically, not morally.

Was that where MAD came from? Mutually Assured Destruction? Oh, that was later. That was later. I'm not convinced, based on my read of history, that any one nation can unilaterally say, oh, we're going to just do nice things and moral and ethical things with this new technology.

Right. I don't... Yes, let's say you do that, but no one else does it, then... What difference does it make? What difference does it make? You know, you've got to play by the same rule book, but we know that's not likely to happen. I mean, what was interesting... The history of our species offers great evidence for that impossibility.

But when you listen to Batya talking, there's such a strength in the points that she makes you would hope that when people will go you know what yeah and the majority come online and then these guys sit in isolation testing you know intercontinental ballistic missiles but but the mad concept yeah mutual assured destruction right just think about that that That brought the United States and the Soviet Union to the table. Yes. Not because they thought nuclear weapons were bad, but they realized they couldn't win.

Right. And that's the problem. The war. When you can't win. That doesn't mean they weren't thinking about it. Or if they could win, they would. And it also doesn't mean that they've taken into account for what I call the Nero scenario What's that? So what did Nero do he fiddle while Rome burned? Oh, he did he burn it down. He didn't care So what happens if you know, you're still in a position where the danger is

Ever present so just because I spent enough time hanging around military people I don't talk about I'm not talking about Hawks that you know that just what I'm just talking about people who think about the history of conflict in this world right behavior of other members of our species not just one guy standing there going just smell that son that smell do you it? I know where that came from. Yeah, the apocalypse. Apocalypse now. You speak to the generals and the majors. You find invariably they're students of war.

They've understood strategies, they understood histories, the provocations and the outcomes. And most of them are not the warmongers we stereotype them to be. Exactly, because of that knowledge, that understanding. Correct. And so I just, I don't have the confidence. I mean, I wish I was as hopeful as Batya is. I want to be that hopeful.

I will aspire to be that hopeful. So I just wonder when she talks, how far ahead of a story in terms of a technology's development are they? And how far are they playing catch up? And, you know, are they being able to bake it in from the get-go or are they just trying to retro-engineer what's gone wrong? It could be a new emergent philosophy where everyone knows to bake it in from the beginning. That would be a shift in our conduct and in our awareness. Absolutely. The kind of shift, for example, dare I harp on this yet again,

that when we went to the moon to explore the moon, we looked back and discovered Earth for the first time. Yes. Around the world, people started thinking about Earth as a planet, Earth as a holistic earth as a holistic Entity that has interdependent elements There's no one island distinct from the rest right anything else that's going on on this planet. There's no boundaries no boundaries yeah, we share the same air molecules water molecules and that was a a

firmware upgrade to our sensibilities of our relationship with nature. And that's why to this day, people all around the world say, we got to save Earth. Nobody was saying that before we went to the moon and looked at Earth in the sky. All the peaceniks at the time in the 1960s, they were just anti-war. They weren't, let's save the Earth. Nobody had that kind of sensibility. So maybe it's a sensibility upgrade that's waiting to happen on civilization, lest we all die at the hands of our own discoveries. Yeah, I'm going with the last part. I'm just saying that, you know, you talk about Earth Day,

you talk about we went to the moon, and there are people who think we didn't go to the moon and that the Earth is flat. Yeah, we're screwed. And by the Earth Day, the first Earth Day was 1970. Right. While we were going to the moon. And the irony is...

Could have been 1960, but it wasn't. No. Might have been 1980. No, while we were going to the moon, first Earth Day. So is the irony that we lean into AI to get it to help us create ethical and safety architecture. Help it save us from ourselves. I like

that. Maybe that's the way to flip the table. And that should be it. And say, AI, they're bad actors among humans who are trying to use AI to get rid of humans. Now kill them. No!

Chuck! This is where they live. Here's their dress. Docs them. This is their daily routine. Google knows your daily routine. We really are. Android knows what you've been Googling.

It knows everything. We really are. Android. That is for sure. It knows we've been Googling. That's for sure. It knows everything. We really are bad people. Yeah. We really are. Maybe it's the good AI against, that's the future battle. Right. Good AI against. Versus bad AI.

Evil AI. Evil AI. But then again, the bad AI. With a goatee. The bad AI will tell you that the good AI is the bad AI. And then the first casualty of war is always the truth. Ooh. Yeah, well,

Thank you. I don't know who authored that, but that's brilliant. That was deep. Yeah. Yep. Ooh, that's deep.

And truthful. I wish it weren't true. Exactly. Stop speaking the truth. Why don't you lie to us every now and then? Like everybody else. You got to do.

You can give me a new program. All right. This has been our Future of Life installment of StarTalk Special Edition. Yeah. Yeah, I enjoyed this. Yeah, and congratulations to the award winners. They are the people that we need out there.

Yes, lest we not be around to even think about that problem. Absolutely. All right. Gary, Chuck. Pleasure. Always good to have you. Neil deGrasse Tyson here, as always. not be around to even think about that problem in the first place. All right. Gary Chuck. Pleasure. Always good to have you. Neil deGrasse Tyson here, as always, bidding you to keep looking up.