Big Technology Podcast - What the Ex-OpenAI Safety Employees Are Worried About — With William Saunders and Lawrence Lessig

Episode Date: July 3, 2024

William Saunders is an ex-OpenAI Superallignment team member. Lawrence Lessig is a professor of Law and Leadership at Harvard Law School. The two come on to discuss what's troubling ex-OpenAI safety team members. We discuss whether the Saudners' former team saw something secret and damning inside OpenAI, or whether it was a general cultural issue. And then, we talk about the 'Right to Warn' a policy that would give AI insiders a right to share concerning developments with third parties without fear of reprisal. Tune in for a revealing look into the eye of a storm brewing in the AI community. ---- You can subscribe to Big Technology Premium for 25% off at https://bit.ly/bigtechnology Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: [e-mail]

An ex-open AI super alignment team member joins us to share his concerns about the company's trajectory, along with his lawyer, the Harvard Law Professor Lawrence Lessig, who will shed light on the lack of protections for those who speak out. All that and more is coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We have a great show for you today. We're finally going to speak with some of the people behind some of the concerns you've been hearing about the trajectory of open-eastern. AI, especially with regard to the alignment work within the company or really the super alignment work. So we're joined today by a former member of that super alignment team.

William Saunders is here. Welcome, William. Thanks for having me on. Thanks for being here. And it's my great pleasure to welcome Larry Lessig back to the show. He's a professor of law and leadership at Harvard Law School. And he's also representing William Pro Bono here as he goes. And I guess many of his colleagues as well as he goes and speaks out about these issues. Welcome, Larry. Great to be back.

Let's begin just talking a little bit about the vibe within Open AI. William, you left a few months ago. Take us a little bit inside the company so we can understand the environment from which you're coming out of. During my three years at Open AI, I would sometimes ask myself a question. Was the path that Open AI was on more like the Apollo program or more like the Titanic? And, you know, the Apollo program was about like carefully predicting, at assessing risks in doing groundbreaking science, building in enough safeguards to be able to successfully bring astronauts to the moon. And then even when big problems happened, like Apollo 13,

they had enough sort of like redundancy and were able to adapt to the situation in order to bring everyone back safely. Whereas the Titanic, you know, came out of this competitive race between companies to keep building bigger and bigger ships and sort of ships that were bigger than the regulations had been designed for. Lots of work went into making the ship safe and, you know, building watertight compartments so that they could say that it was unsinkable. But at the same time, there weren't enough life votes for everyone. And so when disaster struck, you know, a lot of people died. And OpenAI claimed that their mission was to build safe and beneficial AGI. And I thought that this would mean that they would prioritize, you know, putting safety first. But over time,

it started to really feel like the decisions being made by leadership were more like the White Star Line building the Titanic, prioritizing, getting out newer, shinier products, than, you know, really feeling like NASA during the days of the Apollo program. And I really didn't want to end up working on the Titanic of AI. And so that's why I resigned. It's kind of interesting that you use those examples and not the Manhattan Project, which is kind of the one that's been people have brought up, like the power and the destruction potential of nuclear energy has been something that's been talked about. And I don't know if Sam has compared himself to Oppenheimer, but I've definitely heard

some people make those comparisons. Why did you shy away from that? As long as we're going through the analogy lens, why did you shy away from that one? I think that is another valid analogy. Um, you know, I think this example of the Titanic, you know, makes it sort of clear that there were like, again, like a, uh, you know, safety decisions that could have been made better or something. Right. I think the Manhattan project is more the analogy for like the scope of impact that this technology could have. The companies are claiming this technology will have and are raising, you know, billions and billions of dollars.

based on this premise of the scope of impact. I think it's also a tale of sort of scientists who set out building a technology wanting to do something good in the world, right? The reason the Manhattan Project got started is because scientists looked at what was coming, what was possible, and were terrified that, you know, Adolf Hitler would get the bomb, right? And that this would be, you know, absolutely terrible for the world. And that's why they went to the Americans. Somewhere along the way, you know, at some point, like Hitler, you know, was dead.

Germany had surrendered, and yet the project went on. And again, that's another situation that, like, I would really not like to find myself in. Right. And I was going to ask you whether you think OpenAI is a product or a research company. And obviously, it's both. But the question is what leads? And reading between the lines or maybe just hearing you explicitly, your belief is it's a product company. Is that right?

It's a bit different from just trying to make the products that are most useful today. It's coupled with a vision of the research for how to build towards something called AGI or artificial general intelligence, which is building systems that are as smart as most humans and can do most economically valuable work that humans can do, aka can do most jobs of people. And so it's like the combination of these visions or something is something that I'm more concerned about. It's where that they're like building on a trajectory where they are going to do, like, AGI as stated, will be a tremendous change to the world. And so they're on this trajectory to like change the world. And yet when, you know, they release things, their priorities are more like a product company. And I think that is like what is most unsettling. You came out with this letter recently talking about how there needs to be a right for people within these companies to warn the public about some of the concerns they have.

Here's a quote from the letter. There is no effective government oversight of these corporations. Current and former employees are among the few people who can hold them accountable to the public. yet broad confidentiality agreements block us from voicing our concerns. So we're going to get into those agreements in a bit with Larry, but you obviously signed the agreement. You're here talking today. The main question I think the public has from you and the people who have signed this letter is, did you see something? Like, did you see something concerning enough inside the company to merit speaking out here? And if so, what was it?

To set something's clear. If there was, if there was a group of people, that I knew were being seriously harmed by this technology. First, I still really hope that open AI would like do the right thing and address this if this was like a very clear-cut case. I also personally would, you know, ignore any sort of like considerations of how I might be retaliated against, and I would talk about that plainly. So that's not what I was seeing. It's also, I don't think that I was working on the Titanic. I don't think that GPT4 was the Titanic.

I'm more am afraid that like GPT5 or GPT6 or GPT7 might be the Titanic in this analogy. And so I can talk about maybe like a couple of areas here. So one is the former colleague on the super alignment team, Leopold, Aschenbrenner, has like talked on a podcast about how he was like looking, you know, asking some questions about like how the internal security was working at the company. And then, you know, wrote a document containing some concerns. He shared this around. He then got reprimanded because, you know, some of the concerns offended some people. And, you know, personally, I would have like written it in a different way. And I think, but the disturbing part of this was that, you know, the only response was

reprimanding the person who raised these concerns, right? It might be reasonable to reprimand the person and then be like, okay, but these parts of these concerns, we're taking these seriously and we're going to address them. Right. That was not the response. And then he later talked about, like, this was one of the reasons that was offered for him being fired. So I guess what I'm narrowing, I'll let you do the other example in a second, but like what I'm narrowing down on is the people that have raised concerns within the super alignment team, it's not like they've all seen some powerful, dangerous technology that they don't believe open AI, like in the immediate term, they don't believe open AI is going to handle appropriately. It's more of like what is the path in the future and that's where this right to warn thing comes down. I'll say here at the beginning, we're going to get into right to warn. I'm fully in favor of this right to warn. And I'm really glad that you brought it up. But I think that it's important for the public just to establish that like inside open AI today, there's this group that's just left. It's not like there was this question like, did Ilius see something? Right. Did you guys see something? It's not that there's something immediate and harmful that you've seen. It's more of like

you're concerned about the path that this company can go down. Is that right? Yeah. Yeah. And I think, you know, this right to warn, this is a right to warn responsibly. This is not a right to like cause like unnecessary panic or something. And I am, most of my research was driven by, again, concerns along this trajectory that the companies are going along, have demonstrated progress on, and are raising billions of dollars to keep going along this trajectory. But I do think there could be like, there could be things happening today that we don't know about. And so the sort of scenario that I'm worried about happening today is suppose that there's some group of people that wants to, you know, spread a lot of disinformation on social media.

Let's say they want to manipulate an election or they want to, you know, incite violence against a minority ethnic group. And let's say that they're in a non-English speaking country. And so, you know, again, most of the people at OpenEye speak English. Most of the alignment work is done in English. And so, like, it would be, you know, somewhat harder for the company to notice this. Now, like, the models are, like, safety trained to, like, refuse requests to do things that are inappropriate, you know, or, like, that seemed like they might be harmful. Now, you know, Open AI has, like, caught actors, like, generating disinformation.

And so clearly they were able to either, like, you know, the safety training didn't apply or they were able to bypass it, right? There's this technique of jailbreaking where you, like, change how you frame the request in order to, like, bypass the safety limits. Maybe you just ask it in a different language or you tell some story around it. So now you've got a group of people. They have the ability to get the model to go along, generating lots of disinformation. And then, you know, the other line of defense that you might have here would be monitoring, the company looking at it. And I have, you know, concerns that, like, there might be a lot of ways that monitoring might miss things. Right. So, for example, like some systems that the company has talked about involve using a very, like, small and dumb language model to, like, monitor what a larger language model is doing, and this will, like, clearly mess a bunch of things.

You know, or there are, like, there might be ways to, like, send requests in that, like, go through some pathway that is just not subject to monitoring or, like, you know, the people can't look at what the actual requests and the completions are. just because the company just doesn't store that. And so now you might have a group of people like generating massive amounts of disinformation using like open AI products and you know the company wouldn't know about it. And it's like you know in this in this case if this happens in an English speaking country

you know somebody might notice and eventually tweet about it and the company would like find out through that pathway. Right. But if it's in a non-English speaking country right you know I don't know how big it could get before people would notice and I really would want a company, like, taking this kind of step to, you know, have, you know, this is a scary story, right? Tell me why this can't happen. And, like, you know, have somebody who is outside of the company who can, like, have an independent assessment that can say, like, yes, this can't happen, you know. And then I will, like, be able to rest easy.

But I can't. Yeah, and I want to get to Larry here because we should talk about the broader context here. which is that, first of all, believe it or not, William is the first Open AI employee that we've had on the show in four years. He's the first one expressing criticism of Open AI from within, or like from previously within, for four years. And recently we kind of had a, you know,

at least from the former employee standpoint, we figured out why. And that is because there are these broad nondisclosure agreements that Open AI employees have to sign before they leave. So I think we're going to talk about those first. And then we'll talk a little bit about this, I guess, new regulation or law that you are, you're both advocating for, which is basically a law that's going to allow employees to, or I don't know, a rule that will allow employees to whistleblow, even if there's nothing imminently illegal that's happening within AI. But let's talk about the NDAs first. So when people leave OpenAI, what are they forced to sign?

and why has that made it so difficult for people like William to speak out? Okay, but I want to actually first tag on to something William just said, which I guess I think is really important. Yeah, great. I first got into this space of whistleblowing protection, helping Francis Haugan, who was the Facebook whistleblower. And what William just described, of course, is what happened inside of Facebook with Myanmar Rwungi genocide,

which was basically the technology wasn't able to monitor the hate that was being spread by the government in that country, which led to, you know, tens of thousands of people being murdered. And while this is happening, people in the company are trying to raise the alarms and the company is not willing to devote the resources necessary to address the harm that they are able to demonstrate their company is committing. And the reason why that experience is relevant is it shows exactly why you can't rely on the company alone. When you've got a company in a deeply competitive market that's focused on a, you know, Facebook's had a single dimension,

which is like user engagement. Like, are we continuing to meet our target? And an employee, you know, there are a lot of great engineers in that company who raised concerns that were valid and serious, but if they're inconsistent with the objective of the company, it's not going to do anything about it. And so that's the structure. that's real in Silicon Valley, that you've got to build around.

And that's why what we're talking about, which we'll get to in a second, as you've said, guarantees that any concerns that are raised are not just raised to the company. They're raised to people outside the company who can do something about it. Now, as to what you're bound by, I got connected to this group, this incredible group of Open AI employees, ex-employees. when I read about the struggle that Daniel had gone through. Another ex-open AI employee who didn't sign the agreement. Yeah, who believed as he was leaving that by not signing an agreement,

he was giving up, as New York Times reported, something like $1.7 million in equity. And when I read that, I was like, wow, I mean, I don't know many people who would give up $1.7 million just for the freedom to speak. that's interesting. Like, what is it that you think you need to say? And when he raised that concern, and we began to talk to people in the circle of the company, very quickly the company realized that the agreements they were forcing people to sign were technically just not legal agreements in the state of California. Equity is wage in the state of California. If you earn equity, your vested equity, it's like your wages. And when you leave, they can't say,

oh, here's a bunch of other additional terms you must agree to in order to take what you've already earned. So the non-disbaragement part, any other additional obligations that were demanded were not actually obligations that could be enforced. And right now, the company's in the process of revising and putting together exit packages that are consistent with the law. And I'm optimistic. We're not, nothing settled yet, but I'm not. domestically you get to place that the company's rules are exactly right, that they, you know, they say you're leaving. Remember, you've got secrets. You can't share and don't. But, of course, you're allowed to share secrets with government investigators or people who are doing work with

the government for safety purposes. They're not trying to block any of that. And to the extent they do that, what they're doing is going to be consistent with the law. But we're in a transition right now. and it's not yet fully resolved exactly how much they've accomplished and how much they still needs to be done. It's interesting. So those non-disparagement that employees had to sign or else they could face their equity being clawed back seems like they're both like non-enforceable and potentially being revised, which is very interesting, good news.

I think OpenAI also says that they've never clawed back any equity nor did they ever intend to, but it's making people sign the agreement is strong enough. Yeah, I mean, but that's right, because, I mean, I've spoken to X, employees who've said, look, I've not done X, Y, and Z because I've feared the Clawbacks. So they can say they never enforced it. They didn't need to enforce it to have the effect, which it had for a significant number of people, especially when you've got, you know, people who are being very conscientious about the kind of obligation they're going to accept for themselves or not. And so if they sign something like that, they're going to live up to it. Totally. So it has an effect whether they enforced it or not, and that's the problem with the agreement. Right. And then, so that's once people leave. So the deeper question is what happens if people are inside the company and they see something they don't like and are they able to speak out? Because let me see if I get this right. In a normal whistleblower situation, let's say you're an Enron and you see the company committing tax fraud. That's obviously illegal. You'd be protected on whistleblower status.

statutes. But if you're within, let's say, an open AI and you find that the development of the technology is moving towards artificial general intelligence or superintelligence in a way that you find dangerous, you're not allowed to say anything because we don't have any laws against developing super intelligence. Yeah. So there's actually, so there's two things that go together that's very important in this context. So one thing you're right, there's not a lot of regulations. So there's not an FAA or an FDA sitting on top of the company that has imposed regulations that the company is either living up to or not living up to. But, you know, some agencies like the SEC takes the view that most anything could potentially be the sort of thing

you'd have privilege to complain to the SEC about because it could potentially affect the value of the company. And to the extent it's potentially affecting the value of the company, it raises SEC concerns. So if you you say you're following the following safety regime as the company in order to make sure AGI is safe and then you don't follow that regime. The SEC's view is you can come out and tell the SEC. You can whistleblow to the SEC and the SEC would consider whether that's something to act on. The problem is, this is the second part, engineers inside of companies like this or policy people inside of companies like this need to have confidence that the people they're

talking to know what the hell they're talking about. Right? So it's one thing to imagine, you know, an AI safety institute where you can imagine going to that and talking to people like you, people who have a really good sense of like what the risks are, what the technology is, and explaining here's why you think there's a concern. It's another thing to imagine like calling the SEC and telling the SEC, here are the seven safety related concerns that I have because you're very anxious that they understand it and are able to act on it in the appropriate way. And so that's why this is a kind of unique situation. It's both that there's not adequate regulation. So there's no regulator on the scene and that it's a technical field that doesn't easily open up to like non-technical lawyer types, the sorts that are going to be working at the SEC. And that's why, you know, when I spoke to the employees that I'm representing, it became clear that they wanted to kind of craft something that was different and new. And that's what the structure of the right to mourn is trying to produce.

So are you advocating for both a new regulatory agency and a rule to protect AI whistleblowers? What are you going to try to get at here? Well, my own view, and I won't speak for my clients here, my own view is, yeah, absolutely. There needs to be a regulatory agency that is overseeing this. I'm not sure what the structure of it is. It's kind of academic to talk about it, given the dysfunction of the federal government right now. But yes, other countries are building things like this, and we ought to be doing the same. And if there were such an agency, it itself would have lots of whistleblower protections built in,

and that would maybe obviate a significant chunk of the need for the rule. But the rule that we're talking about is a rule that initially we're trying to get companies to embrace. I think the most interesting part of the right to warn is the third point where it talks about creating a culture of criticism where the company says, look, we want you to criticize us. We want you to tell us what's going wrong. We want to encourage that. We're not going to punish that because that's the way we become the safest kind of company that we can be. And so that's really about the company itself creating that. And then the other part that I think is really critical is that the company says, we agree, if we create, we'll create this structure that says you can

complain to us and to a regulator and to an independent AI like a safety institute. You can do all three of those things, confidentially and anonymously. And if we do that, we expect you will use that channel. And if we don't do that, we acknowledge you can use whatever channel is necessary to make sure that these safety concerns are out there. But that's obviously designed to create a strong incentive for them to to build a channel for warning. But opening, I would say they already have that channel for warning.

So this is what they've said in the press reports. No, they don't. They say they have avenues for employees to express a concern, including an anonymous integrity hotline and a deployment safety board that they run products through. Right, but that's the company alone. Okay. So what I said is it has to be all three of those things together, right? So it's the company and the regulator and the AI Safety Institute.

So that, again, like we saw with Facebook, a lot of complaints were made to Facebook about the safety or the lack of safety of their product, and the company didn't do anything about it. And so the concern here is you need to have external review as well. And that's why the channel has got to be a channel that goes to three of these entities, so that we have some confidence that somebody is going to do something if there's something that has to be done. Well, I'm just curious from your perspective, do you think going to the SEC like Larry described is something that you or your colleagues would consider, given, you know, if there were things that you saw that didn't sort of hold to the safety protocols that opening I had lined out, or is that a non-starter?

Yeah. So what I would really want, if I went to whistleblow, is to, you know, have, like, somebody on the other end of the phone or the other end of the message line who I know really understands the technology. And I don't know to, you know, who at the SEC would be the person who would really understand the technology. I think that, like, a model that I think might work, you know, I think I personally think would work better would be like the model more proposed in California, Senate Bill 1047, where there would be like a, you know, where the law would create, like, you know, the office of the California Attorney General as a place where you could submit whistleblower complaints to.

and you could have, like, you know, if you have employees who understood the technology there and you could talk to them, you know, and ideally, this doesn't need to be, like, ideally this is not a high-stakes conversation. Ideally, you can just, like, call up somebody at the government and say, like, hey, I think this might be going, like, a little bit wrong. What do you think about it? And, like, talk to them, and they can gather the information. And then hopefully they say, like, okay, this isn't actually that bad, you know? And then you can, like, get on with your day. I think the thing to fight for here is like being able to really like, you know, talk about things before they become big problems. And in that circumstance, the SEC is insufficient.

Going to the SEC sounds like very intimidating. And, you know, it sounds like the sort of thing one would only do, you know, like, you know, it would be, again, it would be better to be like, you know, know, again, like, be able to talk to somebody in some agency who understands the technology and understands, you know, what the safety like system should look like. Great. Well, I have a few more questions that build off what some of the former colleagues of William have said within OpenAI. And then more about the nature of the company and where we might be heading.

So let's do that right after this. Hey, everyone. Let me tell you about the Hustle Daily Show, a podcast filled with business, tech news, and original stories to keep you in the loop on what's trending. More than 2 million professionals read The Hustle's daily email for its irreverent and informative takes on business and tech news. Now, they have a daily podcast called The Hustle Daily Show, where their team of writers break down the biggest business headlines in 15 minutes or less and explain why you should care about them. So, search for The Hustle Daily Show and your favorite podcast app, like the one

you're using right now. And we're back here on Big Technology Podcast with William Sounders. He's a former open AI super alignment team member now here with us expressing his concerns. William, I can't thank you enough for being here and being open about this stuff. And we're also here with Larry Lessig, the professor of law and leadership at Harvard Law School, also representing William and some of his former colleagues. So here's like a couple questions that have come up in discussions of this after you've gone public. So let me start with this one. So Jan Lika, who used to run the Open AI Super Alignment team, which you were on. He said that safety, culture, and processes have taken a backseat to shiny products within Open AI.

We've discussed that already here. So there's an argument that's being made online, and I'm just going to put it out there and would love to hear your thoughts on this, William. That basically the argument is that the group, the superalignment group didn't really see anything. and that the company doesn't really expect to see anything super dangerous for a while. And so it's reasonably putting like the 20% of compute that I was going to give to the super alignment team toward product until the time comes where it makes sense to shift that resources back to alignment work. What do you think about that? Again, I don't think the super alignment team saw like, you know, this is a catastrophe and it's like endangering people now. I think what we were seeing is a trajectory that the company is raising billions of dollars to go down

that leads to somewhere with predictable, unsolved technical problems. Like, how do you supervise something that's smarter than you? How do you make a model that can't be jailbroken to do whatever any unethical user wants it to do? And more fundamentally behind this, you know, how do we understand what's going on inside of these? language models, which is what I was working on, you know, for the second part of my career. And I was leading, you know, a team of four people doing this interpretability research. And like, we just fundamentally don't know how they, how they work inside, unlike, you know, any other technology known to man.

And, you know, there's a, there's a research community that is like trying to figure this out and we're making progress. But I'm, like, terrified that we're not going to make progress, you know, fast enough before we have something dangerous. And, you know, what people were talking about at the company in terms of timelines to something dangerous were, like, there were people talking, a lot of people talking about similar things to, like, the predictions of, like, Leopold Ashenbrenner, where it's, like, three years towards, like, you know, wildly transformative AGI. And so, I think, you know, when the company is, like, talking about this, I think that they have a duty to put in the work to prepare. for that. And when, you know, the super alignment team formed and the computer commitment was evade, you know, I thought that, like, maybe they were finally going to take that seriously.

And we could finally, like, get together and figure out the, like, you know, I could concentrate on the hard technical problems we're going to need to get right before we have something truly dangerous. But, you know, that's not what happened. Some people say that this, conversations like this are kind of doing open AI's marketing work for it. That basically, like, if this technology could potentially like level cities within a few years then like i don't know mackenzie is going to definitely get in there and try to contract with gpt4 what do you think about that conversation i certainly don't feel like what i'm saying here is doing marketing for open AI. I think, you know, we need to be able to have like a serious and sober conversation

about the risks. And risks are not certainties. There's a lot of uncertainty about what could happen. But when you are uncertain about what should happen, you should be preparing for worst case scenarios, right? The best time to prepare for COVID was not when like it had spread everywhere, but when you could start seeing it spreading and you could be like there's a significant chance that it will continue spreading. So this is for both you and Larry. So Joshua Aachiam, who's open AI employee currently, he sort of took issue with the letter on a couple of areas.

I'm just going to read from a tweet thread that he put out there. He said the disclosure of confidential information from Frontier Labs, however well-intentioned, can be outright dangerous. This letter asks for a policy that would, in effect, give safety staff carte blanche to make disclosures at will, based on their own judgment. And he says, I think this is obviously crazy. The letter didn't have to ask for a policy so arbitrarily broad and so underdefined,

something narrowly scoped around discussions of risk without confidential material would have been perfectly sufficient. What do you think about that? So what's interesting about that is I think it means that he didn't actually read the full agreement, right to warn that we were talking about. because the right to warn we were talking about actually talked about creating an incentive so that no confidential information would be released to the public. If they had this structure, imagine a portal again, where you can connect with the company

and with a regulator and with something like an AI safety institute together, the deal was that's what you would use and you wouldn't be putting any information out in the public. The only way that you would, the right to warn asks for recognition of the right to speak to the public if that is if that does not exist. So I, when I read that, I was like, wow, it's missing the most important part, which is an incentive to build something that doesn't require information as released to the public so long as there's adequate alternative channel for that information to flow. And what about this idea that getting something like this established might keep safety staff out of product meetings? Here's again from Joshua at GM. He says,

good luck getting product staff to add you to meetings and involve you in sensitive discussions if you hold up a flag that says, I will scuttle your launch or talk shit about it later if I feel morally obligated. I mean, that's like, I guess, sort of traditional Silicon Valley thinking, but I'm curious what you both think about that. This is not something that I want to achieve. This is a right that should be used responsibly. And so that if, you know, you're involved in decision-making and you feel like you don't, you disagree with the outcome, but you feel like a good faith process is followed. You know, you should be willing to, you should be willing to respect that. And I think nailing this down where it, you know,

getting the right balance of, you know, the legal rights on this is going to be tricky. And, you know, I want to get that right, but this is more like starting a conversation of where it should be. And I think that like, you know, again, I think on the other side, companies shouldn't have, you know, carte blanche to like declare any information about possible harms confidential. But yeah, it's going to be any implementation of this, you know, is going to get more detailed and more nuance trying to defend both, you know, the company's legitimate rights to, you know, confidential information that, like, preserves their competitiveness and also the, like, rights of employees to, you know, warn the public when something is going wrong.

The other thing is, I mean, even if there is the dynamic that you described, there's also so within a company, there's also a dynamic between companies. So if a company were to embrace the right to warn the way we've discussed it, there would be a lot of people like William or others who would say, that's the kind of company I want to work for. And so that company would achieve an advantage of talent that might swamp any cost that they're paying because they're being anxious about who they're sharing safety concerns with, number one. Number two, you know, again, inside of Facebook,

of course there were people inside of Facebook because I don't care what we're doing. I don't care what the world is suffering because of what are doing? You know, what do I care about 10,000 people dying in a country? I've never heard of it. Yeah, they're mostly not like that, though. they're mostly not like that. These are really smart, decent people who went to work for these companies because they're trying to make the world better, especially AI companies.

People who went to work for Open AI at the beginning didn't even have any conception of what open AI was going to be like today. The idea that it made the progress it did was a surprise to most people. So these are the very best motivated people that you could imagine. And I'm not worried that you're going to have a bunch of people who are like, I don't care what we do to the world. We're just trying to make sure our stock achieves its maximum return. Right. And so obviously this will take some buy-in from the top of companies. And I mean, this is a particularly interesting one with Sam Altman at the head of Open AI.

So Sam, you know, he has talked often about how he cares about AI safety. There have been some interesting quotes from him. Like early on, he's like, I think there's a good chance that AI is going to wipe us out. But in the meantime, there are a lot of companies that can make some money from it. I'm sure I'm misquoting him, but that was the spirit of the quote. And then, William, you spoke with the New York Times, I believe, talking about your view of Sam and oversight. You said, I do, I'm pretty sure this is you, I do think with Sam Altman in particular, he's very uncomfortable with oversight and accountability. I think it's telling that every group that maybe could provide oversight to him, including the board and the safety and security committee,

Sam Altman feels the need to personally be on and nobody can say no to him. So just curious, like, what your message would be to him and sort of what type of leader do you think he is in, you know, in this moment? You know, I think I don't recall the exact words, but I think Sam Altman has also said, like, no one should be trusted with, you know, this much power. I think he then went on to, like, say, like, oh, I don't think that's happening. But, you know, I think my message would really be like, you know if you want people to trust you like you should have real systems of accountability

and oversight that you know um and like not try to avoid that okay can i just one more question about like what it's like inside open ai because this is sort of like been the message that that we've gotten from you and some of your counterparts who have made these, you know, sort of declarations about what's going on. This idea of like that it's shiny products and safety and culture, safety takes a backseat. Like, how does that manifest internally when there are product launches and things like that? Like, how did you see that actually play out? I was mostly not in the part of the company that was like participating in product launches, I was doing this research to prepare for the problems that are like

coming down the road. But I think, you know, what that can look like is the difference between like, you know, we have a fixed launch date and we'll like rearrange everything to meet that versus, you know, when there's a like safety process, like testing how dangerous the systems are or like, you know, putting things together where there is like not enough time to do this before the launch date being willing to move it, you know. And it's, I do think, you know, now with like the GPT40 voice mode, the company did say that they were like, you know, pushing the launch back. But I think, you know, the, again, the real question here is are the people who are doing the safety work and doing the testing for dangerous capabilities,

like are they actually, you know, able to have the time and support to do their job before the launch? And, you know, I think a company can say that they're like pushing something back for safety, but still, like, not have all the work done by that time. Okay. Last question for both of you. So I think we've established that there's no, like, immediate term like threat to society or like let's say like titanic sinking style event that could happen with AI but what's the time frame that you think that these concerns might start to creep in given the trajectory of this technology we've talked a little bit today about how leopold believes like maybe within three years but i'm curious like yeah what the time frame is and then

is there is there like the like the frog boiling in the water problem where like this might only be become a problem when we've sort of become immune to it because we've heard so much about the dangers here, even as like chat GPT will hallucinate very basic details. Yeah. So I think like Leopold talks about some scenario of like you get AI systems that could sort of like be drop in like remote replacements for remote workers, do anything that you could get a remote worker to do. And then you could start applying this to like, you know, the development of more AI technology and like other science and that sort of thing happening within, you know, like the three-year time frame. And then this coming with, you know, a dramatic increase in the amount of risk that you could

have from either, like, you know, misuse. If anyone can hire an unethical biology PhD student, does it then make it a lot easier for, like, you know, nefarious groups to conduct, like, create biological weapons? Or also just do we like start putting these systems everywhere in our business? and decision-making roles, and then we've, like, put them, you know, in place in our society, and then, like, you know, a scenario that I think about is these systems become very good at deceiving and manipulating people in order to increase their own power relative to society at large and even, you know, the people who are, like, running these companies.

And I'm, you know, not as convinced about Leopold that this is necessarily going to come soon, But I think, you know, there's maybe like a 10% probability that this happens within three years. And then I think in this situation, it is unconscionable to race towards this without doing your best to prepare and get things right. Yeah, and I would add to that by just reflecting on the cultural difference between people who are in the business of setting up regulatory infrastructures to address safety concerns in general. and people who are in this industry. So when people in this industry are saying, look, between three and five years, it's probably a 10, maybe 20, maybe 30% chance,

we're going to have AGI like capabilities, and that's going to create all sorts of risks. In the safety culture world, you know, outside of these tech companies, three to five years is the time it takes just to even understand that there's a problem, right? So anybody who expects you're going to set up an infrastructure of safety regulation in three to five years just doesn't understand how Washington or the real world works, right? So this is why I feel anxious about this. It's not that I'm worried that

in three to five years everything's going to blow up. It's just that I'm convinced that it takes 10 years to get to a place that we have an infrastructure of regulation that we can count on. And if we're talking about 10 years, what is the real estimate of this technology manifesting these very dangerous characteristics seems to be from what people on the inside are saying pretty significant. So that's why even if it's not a problem today or tomorrow or next year or the year after, we have to, you know, it's a huge aircraft carrier. We've got to turn and it takes a long time to get it to turn. And that work has got to begin today. And I'll just add that, you know, in the real world with the Titanic, right, you didn't have a regulation

that guaranteed that you have enough lifeboats until the Titanic actually sunk, right? And I am on the side of we should have regulation before the Titanic sinks. I mean, man, all that money to get on that boat. And then no life jacket. Seems brutal. All right. William, thank you so much for coming here, spending the time addressing some of the criticisms and being forthcoming about what your concerns are. I mean, hearing from you after you spent some time when the inside has been illuminating to me, and I think it will be for our listeners as well. So thanks so much for coming on. Thank you. And Larry, always great speaking with you. Thank you for bringing such great analysis to the show every time you're on. And I hope

we can speak again soon. Every time you ask. Thanks for having me. Okay. Thanks so much. All right, everybody. Thanks so much for listening. We'll be back on Friday breaking down in the week's news with Ron John Roy. Until then, hope you take care. And we'll see. See you next time on Big Technology Podcasts.