Y Combinator Startup Podcast - #72 - Miles Brundage and Tim Hwang

Episode Date: April 25, 2018

Miles Brundage is an AI Policy Research Fellow with the Strategic AI Research Center at the Future of Humanity Institute. He is also a PhD candidate in Human and Social Dimensions of Science and Technology at Arizona State University.Miles recently co-authored The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation.Tim Hwang is the Director of the Harvard-MIT Ethics and Governance of AI Initiative. He is also a Visiting Associate at the Oxford Internet Institute and a Fellow at the Knight-Stanford Project on Democracy and the Internet. This is Tim's second time on the podcast; he was also on episode 11.The YC podcast is hosted by Craig Cannon.

Hey, how's it going? This is Craig Cannon, and you're listening to Y Combinators podcast. Today's episode is with Miles Brundage and Tim Wong. Miles is an AI Policy Research Fellow with the Strategic AI Research Center at the Future of Humanity Institute. He's also a PhD candidate in human and social dimensions of science and technology at Arizona State University. And Tim is the director of the Harvard MIT Ethics and Governance of AI Initiative. He's also a visiting associate at the Oxford Internet Institute and a fellow of the Knight Stanford Project on Democracy and the Internet. And this is Tim's second time on the podcast. He was also on episode 11, and I'll link that one up in the description. All right, here we go. All right, guys, I think the most important and pressing

question is now that cryptocurrency gets all the attention and AI is no longer the hottest thing in technology, how are you dealing with it? Yeah, Ben Hamner of Kaggle had a good line on this. He said something like, great thing about cryptocurrency is people know. no longer ask me about whether there's an AI bubble. And yeah, it's hard to compete with the crypto bubble or phenomenon, whatever you want to call it. I think it's actually, yeah, good development, right? Like, I mean, the history of AI is like all of these winners. And, like, having another hype cycle to kind of balance it out might actually be a good thing.

Yeah, absolutely. Let's talk about your paper to start off miles. Sure. So, yeah, what is it called? And, yeah, where'd you go from that? Yeah, it's called the malicious use of artificial intelligence. and then there's a subtitle like forecasting prevention and mitigation. And it's attempting to be the most comprehensive analysis to date of the various ways

which AI could be deliberately misused, so not just things like bias and lack of fairness and an algorithm that are not necessarily intentional, but deliberately using it for things like fake news generation and, you know, combining AI with drones to carry out terrorist attacks or offensive cybersecurity applications. And the essential argument that we make is that that needs to be taken seriously, the fact that AI is a dual use or even omnibus technology. And that similar to other fields like biotechnology and computer security, we need to think about whether there are norms that account for that. So things like responsible disclosure when you find out about a new vulnerability is something that's pervasive in the computer security community, but hasn't yet been seriously discussed for things like adversarial examples where you might want to say. say, hey, there's this new misuse opportunity or way in which you could fool this like commercial

system that is currently, you know, running driverless cars or whatever. And so there, there should be some more discussion about those sorts of issues. Okay. And so is it going into the technical details or is it kind of a survey of where you think things are now? Yeah. So most of it's a general survey, but then there's like an appendix on different areas like, you know, how to deal with the privacy issues, how to deal with, you know, the robustness issues and, you know, different places to look for lessons. Okay. And so, Tim, have you been focusing on any of the stuff while you've been here at Oxford, or is your work totally unrelated? It's somewhat related, actually.

I mean, I would say that I'm mostly been focusing on what you might think of as a subset of the problems that Miles is working on, where he's sort of saying, look, AI isn't going to be inherently used for good. And in fact, there's lots of intentional ways to use it for bad, right? And one of the things I've been thinking about is the sort of interface between these techniques and the problems of disinformation. and like whether or not you think these techniques will be used to make, you know, ever more believable fakes in the future and what that does to the media ecosystem. So I would say it's like a very particular kind of bad actor use that Miles talking about. And so when you're doing this research for both of these topics, are you digging into actual code? Like, how are you spotting this in the wild? Yeah.

So, I mean, my methodology is really kind of focused on looking at what is the research that's coming out right now and like trying to extrapolate what the uses might be, right? Because I think one of the really interesting things we're seeing in the AI space is that it is becoming more available for people to do, right? Like you've got these cloud services. You know, we've got the tools are like widely available now. And so I think what's really missing is like the ability to kind of figure out like how you do it. Right. Like what is the methodology that you use? And the question is, do you see papers that are coming out saying, hey, we could actually use it for this somewhat disturbing purpose?

And then kind of extrapolating from there to say like, okay, well, what would it mean for it to get used more widely? Yeah, so like reading paper, seeing what the hot areas are and, you know, cases in which some sort of potentially negative or positive application is, you know, on the cusp of getting, you know, just efficient enough to be used by a wide array of people or, you know, the hyperparameter optimization problem is close to being solved or whatever sort of trend that you might see, like, might be assigned that certain technologies are going to be more widely usable, not just by experts, but potentially in, you know, a huge range of applications. For the purpose of this report that I recently wrote, you know, we got a ton of people together, including Tim, at a workshop, and we talked about, you know, technical trends and, you know, had people in, like, cybersecurity and AI and other areas sort of, you know, give their best guesses of what's possible and then prioritize what the risks are and what to do about them. So I think, you know, a lot of, I think often, like, pulling together different disciplines is a good way to think about what's possible.

And then one other thing that I'll point out is that you don't necessarily have to be. have to even look into the technical literature to find, you know, discussion of these sorts of misuse applications today because it's like a hot topic already. So things like deep fakes for, you know, face swapping and pornography is like a huge media issue right now. And that actually happened while we were writing this report. And then we like added something later about it because we, we talk,

we characterize the general issue of, you know, fake videos and, you know, misinformation and AI as making that more scalable because, you know, potentially required. is less expertise. And while we're writing that, this deepfakes thing happens. And it's, you know, democratizing in some sense the ability to like, you know, create fake videos. So it's, you know, it's quite a live issue. Right. And I think there's a really interesting question here, particularly when you think about, like, prediction about like there's the realm of what can be done and then trying to understand like what's likely to actually happen. In fact, this seems to be the really challenging thing. Because there's like lots of terrible uses for almost every technology.

Yeah. Right. But we see certain. uses more prominently than others, right? And I think that's actually where the rubble on this sort of stuff is, and actually is part of this prediction problem. Yeah, and yeah, so that's why you kind of have to, yeah, I mean, first of all, have some humility about, like, you know, what you can predict, like, you know, if it's a fully general purpose or fairly general purpose technology, they can be steered in a bunch of different directions or applied to a bunch of different data sets, then, you know, you should expect it if it's super widely available, a bunch of people are going to find new uses for it. So, I mean, I think that's a reason to sort of look upstream.

at the papers and see like where the technical trends are because then you can say like well uh you know maybe this is not yet ready for prime time for any application or like this is starting to be like fairly general purpose yeah i mean a good question for you miles is whether or not you think that like we'll see like the virtual uses be the ones that happen first versus the physical ones right so some people have said okay well you could use AI to really make uh you know hacking much easier right or we might be able to use it to create these like fakes right which we're already seeing But I'm wondering if those threats kind of evolve in a way that's like different or maybe even earlier than, you know, threats of like, you know, people have talked about like, oh, what happens if like someone build a drone that goes out and uses its algorithms to go hurt people? Yeah.

It's hard to say. I mean, I think one, you know, heuristic that I've used is that, you know, stuff in the physical world is is often harder. And, you know, like, it's both more expensive and less scale. You have to buy actual robots. And then there's often hardware issues that you run into. and the general problem of perception and perception is much harder in the real world than in static data sets.

But, yeah, we're seeing progress. Like, just a few days ago, there were a bunch of cool videos from Skydeo of their autonomous drone for, like, tracking people doing sports and flying around and seems to be pretty good at navigating in forests and things like that. So, you know, maybe technologies like that

are sort of a sign that, you know, they'll be much more both positive and negative uses in the real world. But yeah, I think in terms of, you know, near-term impact, I think, you know, those sorts of things that have those autonomous features still aren't super easy to use for end users outside of, like, a particular domain. So I'm not sure that, like, anyone could just easily, you know, repurpose it to, you know, track a particular person or whatever. I think it's sort of for, for that domain application. And I don't know how expensive it is, but yeah, probably more, probably more expensive than like a $20 drone. Right, right.

Because I think about, like, what is, like, what's like the first harm that comes out of the gate in a really big way? Because I've debated often like, okay, so like say there's a horrible self-driving car like incident that occurs, right? Like maybe that turns society off in general to the whole technology. And like there's a big categorical outlawing of it. So like I'm like, okay, that's kind of not so good. Right. But at the same time, I'm kind of like, okay, well, what if like hacking becomes a lot more prominent in a way that's powered by a machine learning? But like we know that like, I don't know the response to like huge data disclosures or huge data compromises is like actually quite limited public response.

And that seems not so good either is basically like people either over underestimate the risk or underestimate the risk depending on like what happens first. Yeah, yeah. People are starting to get kind of desensitized to the, you know, these mega disclosures. And so maybe they won't even care if there's some, you know, adaptive malware thing that, you know, that we might be like, whoa, that's kind of scary. But it could be that, you know, something truly catastrophic could happen if you sort of combine the scalability of AI and digital. technology in general with like the adaptability of human intelligence for like finding vulnerabilities. If you put those together, you might have like a really bad, you know, cyber incident that will actually like make people be like, whoa, this AI thing. Yeah, so that's

something that worries me a lot. But it's sort of like a moving goalpost on the positive and negative side, right? Like so, you know, news feed, for instance, like you could call that AI to a certain extent, right, as it's feeding you information. People get mad at newsfeed, they don't get mad at AI, right? So, like, the notion that the public would generally turn on something like that seems almost unrealistic, right? Because you want to just point at one thing. Right, right. I mean, I think it is basically, like, what the public thinks about as AI is an AI, right? Like, what we're actually talking about is, like, this weird amalgam of, like, popular culture, some research explanations that make it to the public, you know, all these sorts of things. And there's so much about, like,

what is, what does the public actually think AI even is, which is really relevant to the discussion, right because right like the news feed assuredly is AI right like it uses machine learning it uses the latest machine learning to do what it does we don't really think about it as AI right whereas like the car is like I mean I think a lot of robots kind of fall into this category where even robots that don't involve any machine learning are thought of as AI right and like actually impact the discussion about AI despite not actually being related to it at all in like some absolute sense well then it sort of becomes a design challenge right it's like why these self-driving cars are shaped like little bubbly toys right there's so much less intimidate

when you see it just like bump into like a little ballard on the street here, whatever. But yeah, the robot, like the factory robot, for instance, like those are terrifying to people. But they've always been terrifying to people. There's no difference here. But surely there are positive things that you guys notice. You know, you're going around to these conferences. Like, what questions are people asking you about AI? What is a public concerned about positively and negatively?

So I think there's two things that are really at top of mind that I think keep coming up both in the popular discussion around AI right now and also among like researcher circles. So the first one is the question of like international competition and like what it looks like in the space. So this is the question of like it seems like China's making a lot of moves to really invest in AI in a big way. What does that mean about like these research fields, right? Will like the US and Canada and Europe sort of stay ahead in this game?

Will they fall behind? And what does that mean if you think that like governments are going to see this as like a national security thing? So that's like one issue I hear a lot about. Second one I think is around the issues of like interoperability, right? which I think are really big concern, which is these systems make decisions. Can we render some kind of satisfying explanation

for why they do what they do? And I use the word satisfy specifically there because there's lots of ways of trying to tell how they do what they do, but this question of how you communicate is a whole other issue. And those seem to be like two really big challenges. I'm sure Miles has seen other things too.

Yeah, I mean, there's a lot going on. The whole fat ML community, fairness, accountability and transparency and machine learning. And now there's like fat star, so it's like more general than just machine learning. conference series and broader community has been doing a ton of awesome work on those sorts of issues. But in addition to the transparency thing that Tim mentioned, I would also mention robustness. So that's a huge concern. And pretty much, you know, if you look at the like offense and

defense in competitions on adversarial examples, like the offense generally wins. Like we don't really know how to make neural nets robust against deliberate or even unintentional things that could mess them up. Like, you know, they do really well according to, you know, one single number of, like, you know, human versus AI performance, but then if it's slightly outside the distribution, they might fail or if, you know, someone's deliberately tampering with it. So that's a huge problem for actually applying these systems in the real world. And, and I think, you know, we'll continue to see progress on that, but we'll also see setbacks where people say, well, this, this proposal you had for, for defending, you know, neural nuts actually doesn't work. And then there are all sorts of

other things besides just adversarial exam. examples, like, you know, there was a recent paper called bad nets that talked about, like, backdoors in neural network. So essentially, like, someone can put a trained neural network on GitHub or wherever. And then, you know, it seems to work fine. But then, like, you know, you show it some special image and then it goes wrong. So, yeah, there are issues around that. In terms of positive applications, one area that is super exciting and that it's, there's so much work on it that I've had to, like, sort of, you know, take a step back and, like, not even try to, like tweet all the interesting stuff that I see on it is health.

So there's like pretty much every day on archive, there's a new paper that's like, you know, superhuman performance on, you know, this, you know, dermatology task or this like, you know, this esophical cancer task. So there's like a ton of activity in that space. And is that specific to, for instance, like image recognition, like CT scan type stuff?

There's a lot of image recognition. I think that's like kind of the low hanging fruit because there's all this progress in image recognition and like things like adversarial examples aren't necessarily a problem in that domain. Like you're hoping that a patient isn't like fiddling with their image or like putting, you know, a little turtle on their chest when they're getting scanned and then it like gives the wrong answer.

So, so yeah, there's tons of applications there, but there's also just more general machine learning stuff like predicting, you know, people relapsing and like having to come back to the hospital and like when's the optimal time to like, you know, send people home or like given this huge data set of people's, you know, medical histories, what's the, the best diagnosis. So there's a lot other, a lot of other applications. Yeah, there's a workshop at NIPS a few years back, was it two years ago that was basically like AI in the wild, I think

was the name of it. And I think that's like a really good way of framing up a lot of the issues that we're seeing right now is like we're moving out of the lab in some sense where it's like, okay, the old task used to be just like, could we optimize this algorithm to kind of do this thing better? But like now there's a bunch of like research trying to figure out like, what do we do when we confront like the practical problems of deploying these things like in the world?

And that links a lot of the interpability stuff. It links a lot of the safety stuff. It links these questions that are specific to health. I think all these come out of a fact that the technology is really finally becoming practical. And so you have to solve some of these really practical questions. And so as far as deploying this stuff in the wild in the health use case, like, who is using it right now? Where are we seeing it? A lot of it's pilot stuff.

So like, you know, there'll be a hospital here, you know, medical center there. I am not sure of, you know, any super widely deployed ones, except for, like, like apps for very specific things like, you know, looking at skin lesions and stuff. But yeah, as I said, it's something that's like so active that like I'm not the best person to ask because it's just like, I like haven't even, you know, tried to like, you know, assess what's the hottest thing in this area. This is just like every day. There's like, oh, new pilot on this.

But a lot of it, you know, as Tim said, is like at the stage where it might get rolled out, but it hasn't yet been rolled out. So they're like pilots on the one hand, but then there's also a lot of stuff that's just training on offline data. And they're like, well, if we had implemented this, it would have been good. But, you know, there are issues around interpretability and, you know, fairness and stuff like that that would, you know, have to be resolved before it was actually widely deployed. Right.

I mean, one of the interpretability debates that I'm loving right now is basically, so Zach Lipton, this machine learning researcher did this great paper called The Doctor Just Won't Accept that, right? And it's basically a reference to that trope in a lot of the discussions where it's like, well, the doctor won't accept that it's like not interpretable. Like, what do you mean it's not interpretable? And like, he's challenged. I think what is like a really big question, right? Which is like will they care in the end? Like will interpretability actually matter in the end? And like are we actually in some ways is like the field actually like, you know, over indexing on that or maybe in the very least not thinking as nuanced as it should be about like what kinds of interpability are actually needed or expected in the space. And I think that's like one big question is just like, you know, will these things become the norm for the technology or will, you know, the market kind of adopt it even without.

out those things. And I think if you're worried about the safety of these technologies, that ends of being a question not just of like, can we develop the methods? But can they be something that's just like expected that you use when you deploy the technology? Because it's possible that if you just sort of leave it to the market, that we'll just kind of rush ahead without actually working on these problems. Think about anything, right? Like, do you know how to build a microphone? Like, yet you're totally fine using it. All of these things. And like, you probably see it with like, you know, anti-vaxxers. They're like, I don't know. They're like the old school homegrown version maybe that they don't want to accept it, but the rest of the world seems totally fine with it.

Yeah, and just another point, I think they're likely to be differences cross nationally, not just like intranational in terms of who's going to be willing to accept what. Because, you know, countries in the European Union might be like much more. And at the EU level, there might be a lot more regulation of these sorts of things. You know, there's this whole discussion around right to an explanation and the general data protection regime. In China, there's like much, or I haven't seen as much concern about interpretability, though there are some, like, good papers coming out of China. But in terms of, like, governance, I haven't gotten the sense that they're going to like hold back the deployment

of these technologies for those reasons. And then in the U.S. maybe it's like somewhere between the two. I mean, it's a real battle of like, I was reflecting on this because I saw a debate on interpretability recently where some researchers were like, no one cares. Let's just roll ahead with this stuff. So just to pause you really quick. Let's define that. Let's define that. that just in case someone is listening who's not like an AI nerd. Yeah, sure. So I think the most colloquial way of talking about it is interpability is kind of the study of the methods that let you understand like why a machine learning system makes the decisions that it does. In other ways, like kind of like an audit to understand how you got this output. That's right.

Exactly. Right. And there's two sets of problems there. One of them is can you actually extract like a meaningful explanation to like technicians? And then there's the other question of just like from a user point of view like, you know, just like a doctor or someone who's not like a domain expert on machine learning being able to understand what's going on. Right. Okay. Right. And the debate, I think, focused on just like, doesn't matter.

Yeah. Right. Because I think there's some machine learning folks who are like, look, if it works, it works. You know, and that's ultimately going to be the way we're going to move ahead on this stuff. And some people say, no, we actually want to have some level of explanation. And I actually kind of got the feeling that in some ways this is sort of like machine learning fighting with the rest of the computer science field. Right. Because like when you're learning CS, it's very much about like, can you figure out like every step of the process?

Right. Interesting. And like, you know, whereas machine learning has always been like empirical in some sense, right? Like in the sense that like we just let what the data tells us train the system. Right. And like those are actually two ways of like knowing the world that are actually debating on this question of interpretability. I mean, it's sort of like statistical significance in bio. Whereas like, I don't know.

It worked five out of 500 times. Like, therefore it works. Right. This is fine. It's not a computer. Yeah. And so what are people pushing for? like, for instance, you know, we're in the UK now, in the U.S., how are the conversations different?

So, I mean, I think there is certainly very different regimes around like what is sort of expected from explanation, right? Because I think, and this actually stems from some really interesting things about like how the U.S. thinks about privacy and how the Europe thinks about privacy. But I would say in general, I think the U.S. moves on a very case-by-case basis. So the regulatory mode is basically to say, look, in medical, that seems to be a situation. where there's like particularly high risks and like we want to create a bunch of regimes that are specific to medical. Whereas in Europe, I think there's like broader regimes where the frame is, for example, automated decision making.

Right. And the GDPR applies to automated decision making systems, which is very broad. And the actual interpretation will narrow that considerably. But you start from a big kind of category and you narrow it down versus an approach I think we're just taking much more like just starting from the domain that we think is significant. So it's more patchworky, I guess, in that sense. You would agree? Yep, I agree.

Yeah, fantastic. Okay, cool. So I am curious about your PhD. What are you working on? And you're almost done? Yeah, so I'm studying science policy. And the work in my dissertation is on what sorts of methods are useful for AI policy. And, you know, the problem that I pose is that there's so much uncertainty.

Like there's uncertainty, as we were just talking about, about where AI will be applied. But then there's also deep expert disagreement about how long it will take to get certain capabilities like human level AI or even if that's well defined, let alone what happens after. So I'm taking more of a like scenario planning approach. Like let's think about multiple possible scenarios. And I've done some, you know, workshops and I'm trying to understand, you know, is that a

useful tool? And also can we do like, you know, models that sort of express this uncertainty in some sort of formal way. Yeah, and there's a lot of, like, history you've looked into there, too. Yeah. Yeah. Yeah. So, I mean, I think that one way to, yeah, so I mean, people have been talking about AI, AI ethics and AI governance for a long time, but there hasn't been much dialogue between, you know, this world and then the other worlds of, like, you know, science policy and public policy. And, and, you know, one way to think about is that AI is sort of less mature in terms of its, you know, methodological rigor. Like, you know, the best we've sort of come up with is, like, let's do a survey of some experts. Whereas,

And, you know, you look at something like climate change, you know, they not only, like, you know, do surveys of experts, but also, like, synthesize that expertise into, like, an IPCC report that's supposed to be, like, super authoritative and has, you know, error bars for everything and, like, levels of confidence in different statements. They have this whole process. They have, you know, models of different possible futures, given different assumptions. Everything's sort of much better spelled out in terms of, you know, the links between assumptions and policies and scenarios. So I think, you know, I'm trying to take one small step in that direction of like more rigor and more sort of clarity of, you know, what are the actual disagreements. Are you guys, are you familiar with the history of policy? Because I was driving over here with my girlfriend. And she asked, you know, like, has this like policy ecosystem around AI always existed around CS? Like, for instance, you know, when writing started, were people questioning the policy of like, what does this mean? Is this like a new phenomenon given that? you know, you can establish, for lack of a better word, like a personal brand and, like, disseminated out to the world? Or, you know, have there, you know, kind of always been policy

advisors in as many number as you guys, like working directly with governments and companies and stuff like that? Yeah, I don't know about writing. But definitely, or at least no, no record I heard it as a joke around Joe Rogan, actually. Yeah, yeah, yeah. But certainly things like nuclear weapons and nuclear energy and solar energy and coal and, you know, cars. There were people debating the social implications and there were calls for regulation and there were conflicts between, you know, the incumbent interests and the startup innovators. So I think, you know, those sorts of issues are not new. I think what's more new is, as you said, there's like an ability to spread, you know, views more quickly and to have sort of global conversations about these things.

Yeah, I mean, I think it's just sort of linked to the notion of, like, having specialists develop policy at all. Like I think that's like kind of the history of this, right? Which is like when do certain situations become considered so complex as to require someone to be able to like be like, okay, I can become an expert on it and be like the person who's consulted on this topic? And I think a little bit about like what is like the supply of policy and then also like what is the demand for policy.

Right. So like in the nuclear war case, right, like governments have a lot of interest in trying to figure out how we avoid like chucking nuclear bombs at one another. Right. And so, like, suddenly there is a really strong demand. There's also, like, funding. There's all these, like, there's all these reasons for policy people to kind of enter the space. And I think AI is sort of interesting in that it kind of like floats in this median zone right now, right? Where it's sort of like, you see this happen a lot where people like, AI, it seems like a really big deal. But then get into the room and they're like, so what are we doing here exactly? Like, what is, what is policy and AI? And I think that is part of the challenge right now is trying to figure out, like, what are the things that are really valuable to kind of work on? if you think this is going to continue to become like a big issue. Because right now the technology is nascent in a way that we can argue about the relative impact of it at all. And then we can argue about like does it make sense to actually have kind of like policy people working on as a special thing. You guys, I mean, obviously there are a lot of machine learning papers coming out all the time.

But you're very much at the forefront. Like oftentimes I feel like you're sort of like ahead of the curve a little bit like anticipating the needs and demands of a company or of a government. And so like planning head for the future, like, are you just like waiting for data to come? Are you like getting within companies to like see what they're working on? Are you like learning about the hardware? How are you spending your time to figure out what's coming next? Yeah, I mean, a lot of it's just talking to people, talking to people working on hardware and, you know, in industry and academia and like what they're working on. And sort of, you know, I mean, I find it personally helpful to have some sort of predictions or, you know, you know, explicit model of, you know,

of the future and you know, I've written some like blog posts about this, like my forecast for like short term. So like in 2017, I made a bunch of predictions. I found that to be a super useful exercise because then I could say, okay, what was I wrong about? And was there like, were there systematic ways in which I can sort of be better about anticipating the future next time? Yeah. And I think we had asked an interesting question about like what is what is policy expertise? Because it's like different in different situations. Yeah. So imagine like the nuclear case. And actually the nuclear case is pretty interesting, right? Because early on, the experts from a policy perspective also were like the physicists, right?

And like you could imagine that existing actually in a field or in a technical field, which is society is like, okay, what do we do with this technology? And the response is, well, the scientists working on it will tell you about that, right? But AI is sort of interesting in that like there has been kind of the development of a community of people that I think is fairly nascent, which I think suggests to me that like at least two options, right? Like one of them is that like the field could be like the technical. field could be doing more policy stuff but isn't right now. Okay.

So it's an arbitrage? Maybe, yeah. I mean, that's maybe one way of thinking about it. I mean, but there's also like this other question of just like, what are other things that might help to inform the technical research? Okay. Right? Like, I think a lot of my policy work really is like translation work, right?

Where you like talk to policy people who are like, well, I understand like liability. And I'm like, well, you know, this is, it's mixed up because of AI because of ABC reasons, right? And so, like, it's bringing like the technical research to an existing policy discussion. there's also the reverse that happens, right, which is basically, like, researchers being like, what is this fairness thing, right? And you're like, well, it turns out that you can't just create a score for fairness. Like, there's these really interesting things that people have written about. And, like, how do you think about translating that into the machine learning space as well, which is kind of what you can read, like, Fatimel doing? And so I think that that translation role

is like, it's by no means certain, but in the AI space seems to have been like a useful role for people to play. Again, thinking about, like, what is supply, like policy supply and policy demand. Yeah, absolutely. Yeah, I think collaboration. is super important between people interested in the societal questions and the technical questions. And, you know, it's rare not just in AI, but in other cases to, like, have the answer, like, readily available. So with, like, the ICCCC for climate change, like, they have to go back to the lab sometimes and do new studies because they're trying to answer policy relevant questions. So I think AI might be the sort of case where there's sort of this feedback loop between people saying, okay, here are the questions that AI people need to answer. Like, here are the assumptions we need to flesh out, like, in terms of, you know, how quickly,

will we have this capability and so forth, that you can't just find that existing on archive. Like the answers aren't just lying out there ready to be taken by policy people. I think there needs to be this sort of collaboration. Yeah, I'd love to actually look into the history of how this evolved in the climate, like, science space, right? Because you can imagine a situation where, like, you hear this from some machine learning people sometimes, which is like, I just programmed the algorithms, man.

Like, other people have to deal with, like, I don't know, the implications of that, right? And, like, presumably you could actually have that in the climate space as well, where researchers could be like, all I do is really measure the climate. climate, man, like, you decide if you want to change emissions, like, that's not my deal. But clearly, like, that field has taken the choice to basically say, like, in addition to our research work, we have this other obligation, which is to engage in this policy debate. Right. And I think that is really interesting is, like, what does the field actually think its responsibilities even are? And then, like, how do other kind of, like, skills or talents arrange themselves around that? So then the question

ends up being, like, Tim, Tim, you were at Google before. Now we're at the future of Humanity Institute. and how do you guys deal with policy both within an institute and within a company? What are the differences and how do those relationships work? Yeah, definitely. So I've got kind of a weird set of experience, I think, just because I was doing public policy for Google, so that was very much on the company side of things. And then now I'm doing a little bit of work with Harvard and MIT on this ethics and governance of AI initiative and doing work with the Oxford Internet Institute as well.

And it is interesting, like the degree to which, you know, you actually, you find that people in both spaces are often concerned about the same things. But the constraints that they operate under are very different. Right. So, you know, both sides, I think, like, I talked to a bunch of researchers within Google who are, like, very concerned about fairness. I talk to researchers outside of Google who are in civil society, right, who are very concerned about fairness. Have you found the same to be true? Yeah. Yeah. So I think there are people worried about the same issues in a bunch of different domains, but they differ in terms of, you know, how much time they're able to focus on them and what sorts of concrete issues they have to answer. So like if you're in industry,

you have to sort of think about the actual applications that you're rolling out or like, you know, fairness as it relates to this product, you know, assuming that you're working on the application side. They're also researchers who are interested in the more fundamental question. But in terms of, you know, different institutions. And, you know, if you're in government, you might have a broader mandate, but you don't have the time to like drill down into every single issue. You need to sort of rely to some extent on experts outside the government who are, you know, writing reports and things like that. And then if you're in academia, you know, you might be able to take a super broad perspective, but you're not necessarily as close to the, you know, cutting edge research.

And you have to sort of rely on having connections with industry. So, for example, at the Future of Humanity Institute, we have a lot of relationships with organizations like DeepMind and Open AI and others. But, you know, we don't have like a ton of, you know, GPUs or TPUs here, like, running the latest experiments outside of, you know, some specific domains like safety. So yeah, I think, you know, having those different sectors in dialogue is super important in order to like have, have a, you know, synthesis of, you know, what are the actual practical problems we're pressing? What are the governance issues we need to address across this whole thing? And then like, you know, what are the issues we need people to drill down on and focus and like do sort of, you know, free range, you know,

wide ranging exploration of that are like further down the road. And so what does the population look like here of researchers. I'm curious in the sense of, like, who's around, like, influencing your ideas? Like, what are their backgrounds? What are they working on? Yeah, so it's at the future of Humanity Institute, it's a mix of people. So there's some philosophers, an ethicist, there's some political scientists, there's some mathematicians.

And, you know, it's basically a mix of people who are interested in both AI, or not everyone's working on AI, but AI and biotechnology are two, like, technical areas of focus. also more general issues related to the future of humanity, as the name suggests. So it's pretty interdisciplinary. Like people aren't necessarily working just in the domain that they're coming from. So like the mathematicians aren't necessarily, you know, trying to, you know, prove math theorems, but rather just like bringing that mindset of, you know, rigor to to their work and trying to like, you know, break down the concepts that we're thinking about. Yeah, I'm curious about this too, because I've never really understood this about FHA is sort of the argument that like thinking about

existential risk. There's like practices that apply across all these different domains, or do they kind of operate as sort of like separate research? We should pause there too. Is the existential risk at the crux of the FHA being founded? Yeah. So it's a major motivation for a lot of our work. So like the book Superintelligence by our founder, you know, talked a lot about existential risks associated with AI. But it's not the entirety of our focus. So we also are interested in, you know, long term issues that aren't necessarily existential and also making sure that we get to the upsides. So I think I'm ultimately pretty optimistic about the positive applications of AI. So I think we do a range of issues. But yeah, but like to Tim's question, there are a lot of people who come at this from a sort of, you know,

like very conceptual and like utility maximizing, you know, philosophical perspective of like, whoa, if we were to like lose all the possible value in the future, it would be as humanity just stopped. That would be, you know, one of the worst things that could possibly happen. And So reducing the probability of existential risk is super important even if AI is decades or centuries away. And even if we can only decrease the probability of that happening by like 0.1% or whatever in expectation, that's like a huge amount of value that you're protecting. So before we wrap things up, I'm curious about your broad thoughts. Like, what should we be concerned about in the short term around AI and in the long term?

And then how do the two mix together? Yeah, definitely. I mean, so I think this is one of the really interesting things is that, at least within the community of policy people and the kind of researchers, right, that there has been this kind of beef, if we will. I mean, maybe beef is a little dramatic, but a small beef, you know, between, like, what we might call, like, yeah, the long term, like you're talking about, which is, like, people are concerned about AGI and existential risk and all these sorts of things. And then sort of the short term, people saying, like, well, why do we focus on that when there's all these problems of how these systems are being implemented right now? And yeah, I mean, I think that is one of the kind of enduring sort of features of the landscape right now. But I think it's an interesting question as to whether or not that will be, you know, the case forever. I don't know.

Like I know Miles, you've had some thoughts on this. Yeah, yeah. So I think there are common sort of topical issues over different time frame. So like both in the near and the long term, we would want to worry about systems being fair and accountable and transparent. And maybe the methods will be the same or maybe they'll be different over those different time horizons. And I think there are also going to be issues around security over different time horizons. So, yeah, I think that, you know, there's probably more common cause between, you know, the people working on the immediate issues and the long-term issues than is often perceived by some people who see it as, like, a big trade-off between, like, who's going to get funding or, like, you know, this is getting too much attention in the media. But I think actually, you know, the goal of most of the people working in this area is to, like, maximize the benefits of AI and minimize the risks.

And it might turn out that some of the same governance approaches are applicable. Like, it might turn out that setting, that actually solving some of these near-term issues will set a positive precedent for solving the longer ones and start building up a community of practice and links with policymakers and expertise in government. So, yeah, I think there's a lot of opportunity for fusion. Yeah. What I'm interested in, I mean, you're in kind of like the, this kind of safety community. And like, do you hear people talking about like, I mean, I use the phrase fat AGI, which I think is just fascinating as a term. just because it marries together these two concepts so well. Yeah.

But I don't know if that's, is that being talked about at all? Yeah. So I think there's, yeah, there's common cause in the sense that you could sort of, so I mean, so take a step back. So one term that people often throw around in the like AI safety world, particularly looking at long term AI safety is value alignment. So how do you actually learn the values of humans and not, you know, go crazy and do, I mean, you know, to put it colloquially, you know.

That's a technical term in the research. Go crazy. Yeah. Just go crazy old time. Yeah. But I think, you know, you could frame a lot of current issues as value alignment problems, so things around bias and fairness. So I think ultimately, you know, there's a question of how do you extract human preferences

and how do you deal with the fact that humans might not have consistent preferences and some of them are biased. So I think, you know, ultimately those are issues that will have to deal with in the near term and, like, might take a different form in the future if AI systems are operating, you know, with a much larger action space. They're not just like classifying data, but they're, you know, taking, you know,

very long-term decisions and thinking, you know, abstractly. But yeah, I think, you know, ultimately the goal was the same. It's to like get,

you know, the right behavior out of these systems. And that was very interesting because the example that you just gave was saying, you know, a lot of the fairness problems that we're dealing with right now

are actually value alignment problems. Yeah. Which is like the problem there is basically the system doesn't behave in a way that's like consistent with, human values. Yeah, yeah. So that's a fairness case.

And then, you know, so like, you know, that's the F in the fat acronym. I mean, to take accountability and transparency, I think there's also common cause. So, you know, one of the issues I've been toying with recently is that that transparency might be a way of avoiding certain, you know, international conflicts or it might be part of the toolbox. So historically in arms control agreements, like around nuclear weapons and chemical weapons, there have been things like on-site inspections and, you know, satellite. light monitoring and all these tools that are sort of bespoke for the purpose of the domain. But the general concept is we would be better off cooperating and we will verify that that

behavior is actually happening. And so that, you know, if we detect defection by the Soviet Union or the Soviet Union detects defection from us, then they can respond appropriately. But, you know, we can build, you know, trust but verify in Reagan's terminology. And I think if if you actually had the full development of the fat methods and you had accountability and transparency for even general AI systems or super intelligent systems, I think that would open up the door for a lot more collaboration. If you could sort of credibly commit to saying, okay, you know, we're developing this general AI system, but, you know, these are its goals or this is how it learns its goals. And, you know, we're sort of, you know, putting these hard constraints on the system such that

it's not going to attack your country or whatever. Yeah. I think what's, I mean, one of the things it's so intriguing about it though is like the reason why like fat a g i for me is like oh it's like kind of kind of a crazy idea is because i know typically in like the literature around aGI it's very much like the idea that it would be accountable and that it could be transparent is usually considered impossible right because like a g i so complex and so powerful that it would like that nothing could do that but almost yeah i mean the movie you're making the movie you're making is to say like actually we might we might be able to do it right well there are differences of opinion on like how sort of interactive the development of, you know, an AGI would be and, you know, the extent to which humans

will be in the loop, you know, over the long run. And so I mean, Paul Cristiano at OpenAI, for example, has a lot of really good blog posts. And, you know, some of these ideas are in the paper concrete problems in AI safety about, you know, about the idea that, you know, courageability, what he calls corrigibility and what others have called corrugability might actually be like a stable basin of attraction in the sense that if a system, you know, is designed in such a way that it's able to like take critical feedback and it's able to say okay yeah what i was doing was wrong that might sort of like stabilize in a way that it's like continuously asking for human feedback so it's possible that accountability is you know an easier problem even for very powerful systems than we

realize like you know there are powerful uh you know maybe trump aside there are powerful people in the world who actually seek out critical feedback and like are aware uh and like want to hear diverse inputs and like want to make sure that they're doing the right thing right but this is actually really interesting because it's like it's both short term and long term again right which is like if we could get the research community to have certain norms around ensuring that like we are seeking to build corrugible systems yeah that that might set the precedent that the iGI that eventually arrives will be one which is actually consistent with that yeah right versus like not right we actually have control over the design of the eventual thing right i've always had such trouble understanding like

the people who thought there are these AI engineers that were trying to take over the world with their AGI. It's like, no, they're going to die too. Like all the incentives are aligned. You just like imagine this apocalyptic scenario. But do you guys have, you have strong opinions on people working in public versus working in private? I know there's like somewhat of a debate around development. Yeah, so you mean like working the U.S. government versus. No, no.

Sorry. Do you have an opinion on like trying to build an AGI in holding some amount of your data? or training data, like publicly versus private data. Yeah, so that's a super interesting question. And I think, you know, we sort of broach the topic in this report on the malicious uses of AI because I think there might be specific domains in which, you know, maybe it's not, maybe in a world in which, you know, isn't necessarily the world we're in today, but maybe in a world in which, you know, there are millions of driverless cars.

And, you know, they're all using the same, like, convolutional neural net that is, like, vulnerable to this, like, new adversarial example that you just came up with. you might want to, like, give those companies a heads up before you just, like, post out an archive and then someone can, like, cause tens of thousands of car crashes or whatever. So I think, you know, we might want to think about norms around openness in those specific domains where, you know, the idea isn't to, like, never publish, but it's to, like, have some sort of process. But, yeah, as far as general AI and research right now, the community is pretty open. And I think it's sort of both in the, you know, broad interest and in the individual interest of companies to be fairly open because they want to. to recruit researchers and researchers want to publish. So I think, yeah, there's a pretty strong norm around openness, but if we were in a world

where there was, like, more widely perceived, you know, great power competition between countries or where the safety issues were a lot more salient or there were some, like, catastrophic misuses of AI in the cyber arena, then I think people might think twice. And it might be appropriate to think twice if, you know, your concern is that, you know, that the first people to, you know, press the button, if they're not, you know, conscious of all the safety issues could cause a huge problem. Yeah, I'm very pro open publishing. Like, I think, like, it should be the default.

And it's like, I'm still disputing situations where I'm like, you shouldn't publish on this stuff. Just because, like, I think it is actually to the benefit of everybody to know what the current state of the field is, because it allows us to make, like, a realistic assessment, regardless of whether or not you believe in AGI or you believe in superintelligence, like, you know, like, it's useful just to know, like, what can be done. because even if you're thinking about the more prosaic bad actor uses, right?

Like, it's useful to know, like, what are the risks? And we can't do that in an environment where, like, lots of people are kind of holding back. And so it's important to know the state of the field at any given time so we can actually make realistic public policy. Otherwise, we're really operating in the dark. Yeah, that's a great point. Okay, so, Miles, last year you wrote about predictions for 2017

or 2018? Yeah. Yeah, I made the predictions early 2017, and then I reviewed them like a month ago. Okay. This year, 2018. You can get a full year. I was not prepared for this.

You can have a three-year gap, a three-year timeframe then. Even more. Three years. Sure. Yeah, I think there will be superhuman StarCraft and Dota, too, probably in that time horizon. I said in, I think, early 2017, that it would be the end of, that I gave like 50% chance by the end of 2018. So this gives me more runway. I'll say, you know, they're like 70% confident that, you know, that, you know, they'll be superhuman and StarCard.

I'm actually less familiar with Dota, too. So I'll say just StarCraft. All right. Okay. Tim? I think meta-learning will improve significantly. So this is basically treating machine learning, designing machine learning architectures as if they were their own machine learning problem. It's something that basically is done by like machine learning specialists right now.

And the question is how far will machine learning researchers go in replacing themselves, essentially? and I think that will get really good in ways that we don't expect. And your insight into why that will happen is what? There's some of the results that we're seeing from the research right now. It just seems like these networks are able to kind of tune their parameters in a way that, at least I would have not expected. And so it's cool seeing that adapt and advance. These are all positive things. All right, guys, well, thanks for your time.

Cool. Thanks for having us. All right. Thanks for listening. So as always, you can find the transcript and the video. at blog.w.Ycombinator.com. And if you have a second, it would be awesome to give us a rating and review

wherever you find your podcast. See you next time.