#72 - Miles Brundage and Tim Hwang
00:00Hey, how's it going? This is Craig Cannon, and you're listening to Y Combinators podcast. Today'sepisode is with Miles Brundage and Tim Wong. Miles is an AI Policy Research Fellow with the StrategicAI Research Center at the Future of Humanity Institute. He's also a PhD candidate in human andsocial dimensions of science and technology at Arizona State University. And Tim is the director of theHarvard MIT Ethics and Governance of AI Initiative. He's also a visiting associate at the OxfordInternet Institute and a fellow of the Knight Stanford Project on Democracy and the Internet.And this is Tim's second time on the podcast. He was also on episode 11, and I'll link that one upin the description. All right, here we go. All right, guys, I think the most important and pressing
00:43question is now that cryptocurrency gets all the attention and AI is no longer the hottest thingin technology, how are you dealing with it? Yeah, Ben Hamner of Kaggle had a good line on this.He said something like, great thing about cryptocurrency is people know.no longer ask me about whether there's an AI bubble.And yeah, it's hard to compete with the crypto bubble or phenomenon, whatever you want to call it.I think it's actually, yeah, good development, right?Like, I mean, the history of AI is like all of these winners.And, like, having another hype cycle to kind of balance it out might actually be a good thing.
01:18Yeah, absolutely.Let's talk about your paper to start off miles.Sure.So, yeah, what is it called?And, yeah, where'd you go from that?Yeah, it's called the malicious use of artificial intelligence.and then there's a subtitle like forecasting prevention and mitigation.And it's attempting to be the most comprehensive analysis to date of the various ways
01:35which AI could be deliberately misused, so not just things like bias and lack of fairnessand an algorithm that are not necessarily intentional, but deliberately using it for thingslike fake news generation and, you know, combining AI with drones to carry out terrorist attacksor offensive cybersecurity applications.And the essential argument that we make is that that needs to be taken seriously, the fact that AI is a dual use or even omnibus technology.And that similar to other fields like biotechnology and computer security, we need to think about whether there are norms that account for that.So things like responsible disclosure when you find out about a new vulnerability is something that's pervasive in the computer security community, but hasn't yet been seriously discussed for things like adversarial examples where you might want to say.say, hey, there's this new misuse opportunity or way in which you could fool this like commercial
02:28system that is currently, you know, running driverless cars or whatever. And so there, there should besome more discussion about those sorts of issues. Okay. And so is it going into the technicaldetails or is it kind of a survey of where you think things are now? Yeah. So most of it's a generalsurvey, but then there's like an appendix on different areas like, you know, how to deal with the privacyissues, how to deal with, you know, the robustness issues and, you know, different places to look forlessons. Okay.And so, Tim, have you been focusing on any of the stuff while you've been here at Oxford, or is your work totally unrelated?It's somewhat related, actually.
03:01I mean, I would say that I'm mostly been focusing on what you might think of as a subset of the problems that Miles is working on, where he's sort of saying, look, AI isn't going to be inherently used for good.And in fact, there's lots of intentional ways to use it for bad, right?And one of the things I've been thinking about is the sort of interface between these techniques and the problems of disinformation.and like whether or not you think these techniques will be used to make, you know, ever more believable fakes in the future and what that does to the media ecosystem.So I would say it's like a very particular kind of bad actor use that Miles talking about.And so when you're doing this research for both of these topics, are you digging into actual code?Like, how are you spotting this in the wild?Yeah.
03:41So, I mean, my methodology is really kind of focused on looking at what is the research that's coming out right now and like trying to extrapolate what the uses might be, right?Because I think one of the really interesting things we're seeing in the AI space is that it is becoming more available for people to do, right?Like you've got these cloud services.You know, we've got the tools are like widely available now.And so I think what's really missing is like the ability to kind of figure out like how you do it.Right.Like what is the methodology that you use?And the question is, do you see papers that are coming out saying, hey, we could actually use it for this somewhat disturbing purpose?
04:13And then kind of extrapolating from there to say like, okay, well, what would it mean for it to get used more widely?Yeah, so like reading paper, seeing what the hot areas are and, you know, cases in which some sort of potentially negative or positive application is, you know, on the cusp of getting, you know, just efficient enough to be used by a wide array of people or, you know, the hyperparameter optimization problem is close to being solved or whatever sort of trend that you might see, like, might be assigned that certain technologies are going to be more widely usable, not just by experts, but potentially in, you know, a huge range of applications.For the purpose of this report that I recently wrote, you know, we got a ton of people together,including Tim, at a workshop, and we talked about, you know, technical trends and, you know,had people in, like, cybersecurity and AI and other areas sort of, you know, give their bestguesses of what's possible and then prioritize what the risks are and what to do about them.So I think, you know, a lot of, I think often, like, pulling together different disciplines is a goodway to think about what's possible.
05:14And then one other thing that I'll point out is that you don't necessarily have to be.have to even look into the technical literature to find, you know,discussion of these sorts of misuse applications today because it's like a hot topicalready.So things like deep fakes for, you know, face swapping and pornography is like a huge mediaissue right now.And that actually happened while we were writing this report.And then we like added something later about it because we, we talk,
05:39we characterize the general issue of, you know, fake videos and, you know, misinformation andAI as making that more scalable because, you know, potentially required.is less expertise. And while we're writing that, this deepfakes thing happens. And it's, you know,democratizing in some sense the ability to like, you know, create fake videos. So it's, you know,it's quite a live issue. Right. And I think there's a really interesting question here,particularly when you think about, like, prediction about like there's the realm of what can be doneand then trying to understand like what's likely to actually happen. In fact, this seems to be thereally challenging thing. Because there's like lots of terrible uses for almost every technology.
06:17Yeah. Right. But we see certain.uses more prominently than others, right? And I think that's actually where the rubble on this sort ofstuff is, and actually is part of this prediction problem. Yeah, and yeah, so that's why you kind of haveto, yeah, I mean, first of all, have some humility about, like, you know, what you can predict,like, you know, if it's a fully general purpose or fairly general purpose technology, they can besteered in a bunch of different directions or applied to a bunch of different data sets, then,you know, you should expect it if it's super widely available, a bunch of people are going tofind new uses for it. So, I mean, I think that's a reason to sort of look upstream.
06:48at the papers and see like where the technical trends are because then you can say like well uh you knowmaybe this is not yet ready for prime time for any application or like this is starting to be likefairly general purpose yeah i mean a good question for you miles is whether or not you think that likewe'll see like the virtual uses be the ones that happen first versus the physical ones right sosome people have said okay well you could use AI to really make uh you know hacking much easierright or we might be able to use it to create these like fakes right which we're already seeingBut I'm wondering if those threats kind of evolve in a way that's like different or maybe even earlier than, you know, threats of like, you know, people have talked about like, oh, what happens if like someone build a drone that goes out and uses its algorithms to go hurt people?Yeah.
07:31It's hard to say.I mean, I think one, you know, heuristic that I've used is that, you know, stuff in the physical world is is often harder.And, you know, like, it's both more expensive and less scale.You have to buy actual robots.And then there's often hardware issues that you run into.and the general problem of perceptionand perception is much harder in the real worldthan in static data sets.
07:52But, yeah, we're seeing progress.Like, just a few days ago,there were a bunch of cool videos from Skydeoof their autonomous dronefor, like, tracking people doing sportsand flying around and seems to be pretty goodat navigating in forests and things like that.So, you know, maybe technologies like that
08:07are sort of a sign that, you know,they'll be much more both positiveand negative uses in the real world.But yeah, I think in terms of, you know, near-term impact, I think, you know, those sorts of things that have those autonomous features still aren't super easy to use for end users outside of, like, a particular domain.So I'm not sure that, like, anyone could just easily, you know, repurpose it to, you know, track a particular person or whatever.I think it's sort of for, for that domain application.And I don't know how expensive it is, but yeah, probably more, probably more expensive than like a $20 drone.Right, right.
08:40Because I think about, like, what is, like, what's like the first harm that comes out of the gate in a really big way?Because I've debated often like, okay, so like say there's a horrible self-driving car like incident that occurs, right?Like maybe that turns society off in general to the whole technology.And like there's a big categorical outlawing of it.So like I'm like, okay, that's kind of not so good.Right.But at the same time, I'm kind of like, okay, well, what if like hacking becomes a lot more prominent in a way that's powered by a machine learning?But like we know that like, I don't know the response to like huge data disclosures or huge data compromises is like actually quite limited public response.
09:13And that seems not so good either is basically like people either over underestimate the risk or underestimate the risk depending on like what happens first.Yeah, yeah.People are starting to get kind of desensitized to the, you know, these mega disclosures.And so maybe they won't even care if there's some, you know, adaptive malware thing that, you know, that we might be like, whoa, that's kind of scary.But it could be that, you know, something truly catastrophic could happen if you sort of combine the scalability of AI and digital.technology in general with like the adaptability of human intelligence for like findingvulnerabilities. If you put those together, you might have like a really bad, you know,cyber incident that will actually like make people be like, whoa, this AI thing. Yeah, so that's
09:56something that worries me a lot. But it's sort of like a moving goalpost on the positive and negativeside, right? Like so, you know, news feed, for instance, like you could call that AI to a certainextent, right, as it's feeding you information. People get mad at newsfeed, they don't get mad atAI, right? So, like, the notion that the public would generally turn on something like that seemsalmost unrealistic, right? Because you want to just point at one thing. Right, right. I mean, I thinkit is basically, like, what the public thinks about as AI is an AI, right? Like, what we're actuallytalking about is, like, this weird amalgam of, like, popular culture, some research explanations thatmake it to the public, you know, all these sorts of things. And there's so much about, like,
10:33what is, what does the public actually think AI even is, which is really relevant to the discussion,right because right like the news feed assuredly is AI right like it uses machine learning it usesthe latest machine learning to do what it does we don't really think about it as AI right whereaslike the car is like I mean I think a lot of robots kind of fall into this category where even robotsthat don't involve any machine learning are thought of as AI right and like actually impact the discussionabout AI despite not actually being related to it at all in like some absolute sense well then itsort of becomes a design challenge right it's like why these self-driving cars are shaped like littlebubbly toys right there's so much less intimidate
11:08when you see it just like bump into like a little ballard on the street here, whatever.But yeah, the robot, like the factory robot, for instance, like those are terrifying to people.But they've always been terrifying to people.There's no difference here.But surely there are positive things that you guys notice.You know, you're going around to these conferences.Like, what questions are people asking you about AI?What is a public concerned about positively and negatively?
11:33So I think there's two things that are really at top of mind that I think keep coming up both in thepopular discussion around AI right now and also among like researcher circles.So the first one is the question of like international competition and like what it lookslike in the space.So this is the question of like it seems like China's making a lot of moves to really investin AI in a big way.What does that mean about like these research fields, right?Will like the US and Canada and Europe sort of stay ahead in this game?
11:58Will they fall behind?And what does that mean if you think that like governments are going to see this as likea national security thing?So that's like one issue I hear a lot about.Second one I think is around the issues of like interoperability, right?which I think are really big concern,which is these systems make decisions.Can we render some kind of satisfying explanation
12:14for why they do what they do?And I use the word satisfy specifically therebecause there's lots of ways of trying to tellhow they do what they do,but this question of how you communicateis a whole other issue.And those seem to be like two really big challenges.I'm sure Miles has seen other things too.
12:28Yeah, I mean, there's a lot going on.The whole fat ML community, fairness,accountability and transparency and machine learning.And now there's like fat star,so it's like more general than just machine learning.conference series and broader community has been doing a ton of awesome work on those sorts of issues.But in addition to the transparency thing that Tim mentioned, I would also mention robustness.So that's a huge concern. And pretty much, you know, if you look at the like offense and
12:54defense in competitions on adversarial examples, like the offense generally wins. Like we don'treally know how to make neural nets robust against deliberate or even unintentional things that couldmess them up. Like, you know, they do really well according to, you know, one single number of, like,you know, human versus AI performance, but then if it's slightly outside the distribution,they might fail or if, you know, someone's deliberately tampering with it. So that's a huge problemfor actually applying these systems in the real world. And, and I think, you know, we'll continueto see progress on that, but we'll also see setbacks where people say, well, this, this proposal youhad for, for defending, you know, neural nuts actually doesn't work. And then there are all sorts of
13:33other things besides just adversarial exam.examples, like, you know, there was a recent paper called bad nets that talked about, like,backdoors in neural network. So essentially, like, someone can put a trained neural network onGitHub or wherever. And then, you know, it seems to work fine. But then, like, you know, you showit some special image and then it goes wrong. So, yeah, there are issues around that. In termsof positive applications, one area that is super exciting and that it's, there's so much work onit that I've had to, like, sort of, you know, take a step back and, like, not even try to,like tweet all the interesting stuff that I see on it is health.
14:08So there's like pretty much every day on archive,there's a new paper that's like, you know,superhuman performance on, you know, this, you know,dermatology task or this like, you know,this esophical cancer task.So there's like a ton of activity in that space.And is that specific to, for instance, like image recognition,like CT scan type stuff?
14:27There's a lot of image recognition.I think that's like kind of the low hanging fruitbecause there's all this progress in image recognitionand like things like adversarial examples aren't necessarilya problem in that domain.Like you're hoping that a patient isn't like fiddling with their image or like putting,you know, a little turtle on their chest when they're getting scanned and then it like givesthe wrong answer.
14:45So, so yeah, there's tons of applications there, but there's also just more general machinelearning stuff like predicting, you know, people relapsing and like having to come backto the hospital and like when's the optimal time to like, you know, send people home or likegiven this huge data set of people's, you know, medical histories, what's the,the best diagnosis. So there's a lotother, a lot of other applications. Yeah, there's a workshopat NIPS a few years back, was it two yearsago that was basically like AI in the wild, I think
15:15was the name of it. And I think that's like a really good wayof framing up a lot of the issues that we're seeing right nowis like we're moving out of the lab in some sensewhere it's like, okay, the old task used to be just like, could weoptimize this algorithm to kind of do this thing better?But like now there's a bunch of like researchtrying to figure out like, what do we do when we confront like thepractical problems of deploying these things like in the world?
15:35And that links a lot of the interpability stuff.It links a lot of the safety stuff.It links these questions that are specific to health.I think all these come out of a fact that the technology is really finally becoming practical.And so you have to solve some of these really practical questions.And so as far as deploying this stuff in the wild in the health use case, like, who is using it right now?Where are we seeing it?A lot of it's pilot stuff.
15:56So like, you know, there'll be a hospital here, you know, medical center there.I am not sure of, you know, any super widely deployed ones, except for, like,like apps for very specific things like, you know, looking at skin lesions and stuff.But yeah, as I said, it's something that's like so active that like I'm not the bestperson to ask because it's just like, I like haven't even, you know, tried to like, you know,assess what's the hottest thing in this area.This is just like every day.There's like, oh, new pilot on this.
16:24But a lot of it, you know, as Tim said, is like at the stage where it might get rolledout, but it hasn't yet been rolled out.So they're like pilots on the one hand, but then there's also a lot of stuff that's justtraining on offline data.And they're like, well, if we had implemented this, it would have been good.But, you know, there are issues around interpretability and, you know, fairness and stufflike that that would, you know, have to be resolved before it was actually widely deployed.Right.
16:46I mean, one of the interpretability debates that I'm loving right now is basically, so Zach Lipton,this machine learning researcher did this great paper called The Doctor Just Won't Acceptthat, right?And it's basically a reference to that trope in a lot of the discussions where it's like,well, the doctor won't accept that it's like not interpretable.Like, what do you mean it's not interpretable?And like, he's challenged.I think what is like a really big question, right? Which is like will they care in the end? Like will interpretability actually matter in the end? And like are we actually in some ways is like the field actually like, you know, over indexing on that or maybe in the very least not thinking as nuanced as it should be about like what kinds of interpability are actually needed or expected in the space. And I think that's like one big question is just like, you know, will these things become the norm for the technology or will, you know, the market kind of adopt it even without.
17:34out those things. And I think if you're worried about the safety of these technologies, that endsof being a question not just of like, can we develop the methods? But can they be something that'sjust like expected that you use when you deploy the technology? Because it's possible that if youjust sort of leave it to the market, that we'll just kind of rush ahead without actually working onthese problems. Think about anything, right? Like, do you know how to build a microphone?Like, yet you're totally fine using it. All of these things. And like, you probably see itwith like, you know, anti-vaxxers. They're like, I don't know. They're like the old school homegrownversion maybe that they don't want to accept it, but the rest of the world seems totally fine with it.
18:06Yeah, and just another point, I think they're likely to be differences cross nationally, not justlike intranational in terms of who's going to be willing to accept what.Because, you know, countries in the European Union might be like much more.And at the EU level, there might be a lot more regulation of these sorts of things.You know, there's this whole discussion around right to an explanation and the general dataprotection regime. In China, there's like much, or I haven't seen as much concern aboutinterpretability, though there are some, like, good papers coming out of China. But in terms of,like, governance, I haven't gotten the sense that they're going to like hold back the deployment
18:40of these technologies for those reasons. And then in the U.S. maybe it's like somewhere betweenthe two. I mean, it's a real battle of like, I was reflecting on this because I saw a debate oninterpretability recently where some researchers were like, no one cares. Let's just roll ahead with thisstuff. So just to pause you really quick. Let's define that. Let's define that.that just in case someone is listening who's not like an AI nerd. Yeah, sure. So I think the mostcolloquial way of talking about it is interpability is kind of the study of the methods that letyou understand like why a machine learning system makes the decisions that it does.In other ways, like kind of like an audit to understand how you got this output. That's right.
19:15Exactly. Right. And there's two sets of problems there. One of them is can you actually extractlike a meaningful explanation to like technicians? And then there's the other question of justlike from a user point of view like, you know, just like a doctor or someone who's not like a domain experton machine learning being able to understand what's going on.Right.Okay.Right.And the debate, I think, focused on just like, doesn't matter.
19:34Yeah.Right.Because I think there's some machine learning folks who are like, look, if it works, it works.You know, and that's ultimately going to be the way we're going to move ahead on this stuff.And some people say, no, we actually want to have some level of explanation.And I actually kind of got the feeling that in some ways this is sort of like machine learning fighting with the rest of the computer science field.Right.Because like when you're learning CS, it's very much about like, can you figure out like every step of the process?
19:56Right.Interesting.And like, you know, whereas machine learning has always been like empirical in some sense, right?Like in the sense that like we just let what the data tells us train the system.Right.And like those are actually two ways of like knowing the world that are actually debating on this question of interpretability.I mean, it's sort of like statistical significance in bio.Whereas like, I don't know.
20:17It worked five out of 500 times.Like, therefore it works.Right.This is fine.It's not a computer.Yeah.And so what are people pushing for?like, for instance, you know, we're in the UK now, in the U.S., how are the conversations different?
20:31So, I mean, I think there is certainly very different regimes around like what is sort of expected from explanation, right?Because I think, and this actually stems from some really interesting things about like how the U.S. thinks about privacy and how the Europe thinks about privacy.But I would say in general, I think the U.S. moves on a very case-by-case basis.So the regulatory mode is basically to say, look, in medical, that seems to be a situation.where there's like particularly high risks and like we want to create a bunch of regimesthat are specific to medical.Whereas in Europe, I think there's like broader regimes where the frame is, for example,automated decision making.
21:08Right.And the GDPR applies to automated decision making systems, which is very broad.And the actual interpretation will narrow that considerably.But you start from a big kind of category and you narrow it down versus an approach I thinkwe're just taking much more like just starting from the domain that we think is significant.So it's more patchworky, I guess, in that sense.You would agree?Yep, I agree.
21:30Yeah, fantastic.Okay, cool.So I am curious about your PhD.What are you working on?And you're almost done?Yeah, so I'm studying science policy.And the work in my dissertation is on what sorts of methods are useful for AI policy.And, you know, the problem that I pose is that there's so much uncertainty.
21:52Like there's uncertainty, as we were just talking about,about where AI will be applied.But then there's also deep expert disagreement about how long it will take to getcertain capabilities like human level AI or even if that's well defined,let alone what happens after.So I'm taking more of a like scenario planning approach.Like let's think about multiple possible scenarios.And I've done some, you know, workshops and I'm trying to understand, you know, is that a
22:17useful tool?And also can we do like, you know, models that sort of express this uncertainty in some sort offormal way. Yeah, and there's a lot of, like, history you've looked into there, too. Yeah. Yeah. Yeah. So, I mean, I think thatone way to, yeah, so I mean, people have been talking about AI, AI ethics and AI governance for a long time,but there hasn't been much dialogue between, you know, this world and then the other worlds of, like,you know, science policy and public policy. And, and, you know, one way to think about is thatAI is sort of less mature in terms of its, you know, methodological rigor. Like, you know, the best we'vesort of come up with is, like, let's do a survey of some experts. Whereas,
22:53And, you know, you look at something like climate change, you know, they not only, like, you know, do surveys of experts, but also, like, synthesize that expertise into, like, an IPCC report that's supposed to be, like, super authoritative and has, you know, error bars for everything and, like, levels of confidence in different statements.They have this whole process.They have, you know, models of different possible futures, given different assumptions.Everything's sort of much better spelled out in terms of, you know, the links between assumptions and policies and scenarios.So I think, you know, I'm trying to take one small step in that direction of like more rigor and more sort of clarity of, you know, what are the actual disagreements.Are you guys, are you familiar with the history of policy? Because I was driving over here with my girlfriend. And she asked, you know, like, has this like policy ecosystem around AI always existed around CS? Like, for instance, you know, when writing started, were people questioning the policy of like, what does this mean? Is this like a new phenomenon given that?you know, you can establish, for lack of a better word, like a personal brand and, like,disseminated out to the world? Or, you know, have there, you know, kind of always been policy
24:02advisors in as many number as you guys, like working directly with governments and companiesand stuff like that? Yeah, I don't know about writing. But definitely, or at least no, no recordI heard it as a joke around Joe Rogan, actually. Yeah, yeah, yeah. But certainly things like nuclear weapons andnuclear energy and solar energy and coal and, you know, cars. There were people debating the socialimplications and there were calls for regulation and there were conflicts between, you know, the incumbentinterests and the startup innovators. So I think, you know, those sorts of issues are not new.I think what's more new is, as you said, there's like an ability to spread, you know, views morequickly and to have sort of global conversations about these things.
24:47Yeah, I mean, I think it's just sort of linked to the notion of, like,having specialists develop policy at all.Like I think that's like kind of the history of this, right?Which is like when do certain situations become considered so complex as to require someoneto be able to like be like, okay, I can become an expert on it and be like the personwho's consulted on this topic?And I think a little bit about like what is like the supply of policy and then also likewhat is the demand for policy.
25:12Right.So like in the nuclear war case, right, like governments have a lot of interest in trying to figureout how we avoid like chucking nuclear bombs at one another.Right. And so, like, suddenly there is a really strong demand. There's also, like, funding. There's all these, like, there's all these reasons for policy people to kind of enter the space. And I think AI is sort of interesting in that it kind of like floats in this median zone right now, right? Where it's sort of like, you see this happen a lot where people like, AI, it seems like a really big deal. But then get into the room and they're like, so what are we doing here exactly? Like, what is, what is policy and AI? And I think that is part of the challenge right now is trying to figure out, like, what are the things that are really valuable to kind of work on?if you think this is going to continue to become like a big issue.Because right now the technology is nascent in a way that we can argue about the relative impact of it at all.And then we can argue about like does it make sense to actually have kind of like policy people working on as a special thing.You guys, I mean, obviously there are a lot of machine learning papers coming out all the time.
26:09But you're very much at the forefront.Like oftentimes I feel like you're sort of like ahead of the curve a little bit like anticipating the needs and demands of a company or of a government.And so like planning head for the future, like, are you just like waiting for data to come?Are you like getting within companies to like see what they're working on?Are you like learning about the hardware?How are you spending your time to figure out what's coming next?Yeah, I mean, a lot of it's just talking to people, talking to people working on hardware and, you know, in industry and academia and like what they're working on.And sort of, you know, I mean, I find it personally helpful to have some sort of predictions or, you know, you know, explicit model of, you know,
26:48of the future and you know, I've written some like blog posts about this, like my forecast for likeshort term. So like in 2017, I made a bunch of predictions. I found that to be a superuseful exercise because then I could say, okay, what was I wrong about? And was there like,were there systematic ways in which I can sort of be better about anticipating the future nexttime? Yeah. And I think we had asked an interesting question about like what is what is policyexpertise? Because it's like different in different situations. Yeah. So imagine like the nuclear case.And actually the nuclear case is pretty interesting, right? Because early on, the expertsfrom a policy perspective also were like the physicists, right?
27:22And like you could imagine that existing actually in a field or in a technical field,which is society is like, okay, what do we do with this technology?And the response is, well, the scientists working on it will tell you about that, right?But AI is sort of interesting in that like there has been kind of the development of a communityof people that I think is fairly nascent, which I think suggests to me that like at least two options, right?Like one of them is that like the field could be like the technical.field could be doing more policy stuff but isn't right now.Okay.
27:52So it's an arbitrage?Maybe, yeah.I mean, that's maybe one way of thinking about it.I mean, but there's also like this other question of just like, what are other things thatmight help to inform the technical research?Okay.Right?Like, I think a lot of my policy work really is like translation work, right?
28:05Where you like talk to policy people who are like, well, I understand like liability.And I'm like, well, you know, this is, it's mixed up because of AI because of ABC reasons, right?And so, like, it's bringing like the technical research to an existing policy discussion.there's also the reverse that happens, right, which is basically, like, researchers being like,what is this fairness thing, right? And you're like, well, it turns out that you can't justcreate a score for fairness. Like, there's these really interesting things that people have written about.And, like, how do you think about translating that into the machine learning space as well,which is kind of what you can read, like, Fatimel doing? And so I think that that translation role
28:36is like, it's by no means certain, but in the AI space seems to have been like a useful role forpeople to play. Again, thinking about, like, what is supply, like policy supply and policydemand. Yeah, absolutely. Yeah, I think collaboration.is super important between people interested in the societal questions and the technical questions.And, you know, it's rare not just in AI, but in other cases to, like, have the answer, like, readily available.So with, like, the ICCCC for climate change, like, they have to go back to the lab sometimes and do new studies because they're trying to answer policy relevant questions.So I think AI might be the sort of case where there's sort of this feedback loop between people saying, okay, here are the questions that AI people need to answer.Like, here are the assumptions we need to flesh out, like, in terms of, you know, how quickly,
29:17will we have this capability and so forth,that you can't just find that existing on archive.Like the answers aren't just lying out there ready to be taken by policy people.I think there needs to be this sort of collaboration.Yeah, I'd love to actually look into the history of how this evolved in the climate,like, science space, right?Because you can imagine a situation where, like, you hear this from some machine learning people sometimes,which is like, I just programmed the algorithms, man.
29:38Like, other people have to deal with, like, I don't know, the implications of that, right?And, like, presumably you could actually have that in the climate space as well,where researchers could be like, all I do is really measure the climate.climate, man, like, you decide if you want to change emissions, like, that's not my deal. But clearly,like, that field has taken the choice to basically say, like, in addition to our research work,we have this other obligation, which is to engage in this policy debate. Right. And I think that isreally interesting is, like, what does the field actually think its responsibilities even are? And then, like,how do other kind of, like, skills or talents arrange themselves around that? So then the question
30:10ends up being, like, Tim, Tim, you were at Google before. Now we're at the future of Humanity Institute.and how do you guys deal with policy both within an institute and within a company?What are the differences and how do those relationships work?Yeah, definitely.So I've got kind of a weird set of experience, I think, just because I was doing public policyfor Google, so that was very much on the company side of things.And then now I'm doing a little bit of work with Harvard and MIT on this ethics andgovernance of AI initiative and doing work with the Oxford Internet Institute as well.
30:41And it is interesting, like the degree to which, you know, you actually,you find that people in both spaces are often concerned about the same things. But the constraintsthat they operate under are very different. Right. So, you know, both sides, I think, like, I talked toa bunch of researchers within Google who are, like, very concerned about fairness. I talk to researchersoutside of Google who are in civil society, right, who are very concerned about fairness.Have you found the same to be true? Yeah. Yeah. So I think there are people worried about the sameissues in a bunch of different domains, but they differ in terms of, you know, how much time they're able tofocus on them and what sorts of concrete issues they have to answer. So like if you're in industry,
31:19you have to sort of think about the actual applications that you're rolling out or like, you know,fairness as it relates to this product, you know, assuming that you're working on the applicationside. They're also researchers who are interested in the more fundamental question. But in terms of,you know, different institutions. And, you know, if you're in government, you might have a broadermandate, but you don't have the time to like drill down into every single issue. You need to sort of relyto some extent on experts outside the government who are, you know, writing reports and things like that.And then if you're in academia, you know, you might be able to take a super broad perspective,but you're not necessarily as close to the, you know, cutting edge research.
31:54And you have to sort of rely on having connections with industry.So, for example, at the Future of Humanity Institute, we have a lot of relationships with organizations like DeepMind and Open AI and others.But, you know, we don't have like a ton of, you know, GPUs or TPUs here, like, running the latestexperiments outside of, you know, some specific domains like safety. So yeah, I think, you know,having those different sectors in dialogue is super important in order to like have, have a, you know,synthesis of, you know, what are the actual practical problems we're pressing? What are the governanceissues we need to address across this whole thing? And then like, you know, what are the issues weneed people to drill down on and focus and like do sort of, you know, free range, you know,
32:36wide ranging exploration of that are like further down the road. And so what does the populationlook like here of researchers.I'm curious in the sense of, like, who's around, like, influencing your ideas?Like, what are their backgrounds?What are they working on?Yeah, so it's at the future of Humanity Institute, it's a mix of people.So there's some philosophers, an ethicist, there's some political scientists, there'ssome mathematicians.
33:00And, you know, it's basically a mix of people who are interested in both AI, or not everyone'sworking on AI, but AI and biotechnology are two, like, technical areas of focus.also more general issues related to the future of humanity, as the name suggests. So it's prettyinterdisciplinary. Like people aren't necessarily working just in the domain that they're coming from. Solike the mathematicians aren't necessarily, you know, trying to, you know, prove math theorems,but rather just like bringing that mindset of, you know, rigor to to their work and trying to like,you know, break down the concepts that we're thinking about. Yeah, I'm curious about this too,because I've never really understood this about FHA is sort of the argument that like thinking about
33:39existential risk. There's like practices that apply across all these different domains, or do theykind of operate as sort of like separate research? We should pause there too. Is the existential riskat the crux of the FHA being founded? Yeah. So it's a major motivation for a lot of our work. So likethe book Superintelligence by our founder, you know, talked a lot about existential risks associated withAI. But it's not the entirety of our focus. So we also are interested in, you know, long term issues thataren't necessarily existential and also making sure that we get to the upsides. So I think I'm ultimatelypretty optimistic about the positive applications of AI. So I think we do a range of issues. But yeah,but like to Tim's question, there are a lot of people who come at this from a sort of, you know,
34:24like very conceptual and like utility maximizing, you know, philosophical perspective of like,whoa, if we were to like lose all the possible value in the future, it would be as humanityjust stopped. That would be, you know, one of the worst things that could possibly happen. AndSo reducing the probability of existential risk is super important even if AI is decades or centuries away.And even if we can only decrease the probability of that happening by like 0.1% or whatever in expectation,that's like a huge amount of value that you're protecting.So before we wrap things up, I'm curious about your broad thoughts.Like, what should we be concerned about in the short term around AI and in the long term?
35:03And then how do the two mix together?Yeah, definitely.I mean, so I think this is one of the really interesting things is that, at least within the community of policy people and the kind of researchers, right, that there has been this kind of beef, if we will.I mean, maybe beef is a little dramatic, but a small beef, you know, between, like, what we might call, like, yeah, the long term, like you're talking about, which is, like, people are concerned about AGI and existential risk and all these sorts of things.And then sort of the short term, people saying, like, well, why do we focus on that when there's all these problems of how these systems are being implemented right now?And yeah, I mean, I think that is one of the kind of enduring sort of features of the landscape right now.But I think it's an interesting question as to whether or not that will be, you know, the case forever.I don't know.
35:46Like I know Miles, you've had some thoughts on this.Yeah, yeah.So I think there are common sort of topical issues over different time frame.So like both in the near and the long term, we would want to worry about systems being fair and accountable and transparent.And maybe the methods will be the same or maybe they'll be different over those different time horizons.And I think there are also going to be issues around security over different time horizons.So, yeah, I think that, you know, there's probably more common cause between, you know, the people working on the immediate issues and the long-term issues than is often perceived by some people who see it as, like, a big trade-off between, like, who's going to get funding or, like, you know, this is getting too much attention in the media.But I think actually, you know, the goal of most of the people working in this area is to, like, maximize the benefits of AI and minimize the risks.
36:32And it might turn out that some of the same governance approaches are applicable.Like, it might turn out that setting, that actually solving some of these near-term issues will set a positive precedent for solving the longer ones and start building up a community of practice and links with policymakers and expertise in government.So, yeah, I think there's a lot of opportunity for fusion.Yeah.What I'm interested in, I mean, you're in kind of like the, this kind of safety community.And like, do you hear people talking about like, I mean, I use the phrase fat AGI, which I think is just fascinating as a term.just because it marries together these two concepts so well.Yeah.
37:05But I don't know if that's, is that being talked about at all?Yeah.So I think there's, yeah, there's common cause in the sense that you could sort of, so I mean,so take a step back.So one term that people often throw around in the like AI safety world, particularlylooking at long term AI safety is value alignment.So how do you actually learn the values of humans and not, you know, go crazy and do, I mean,you know, to put it colloquially, you know.
37:29That's a technical term in the research.Go crazy.Yeah.Just go crazy old time.Yeah.But I think, you know, you could frame a lot of current issues as value alignment problems,so things around bias and fairness.So I think ultimately, you know, there's a question of how do you extract human preferences
37:46and how do you deal with the fact that humans might not have consistent preferences and someof them are biased.So I think, you know, ultimately those are issues that will have to deal with in the near termand, like, might take a different form in the future if AI systems are operating, you know,with a much larger action space.They're not just like classifying data,but they're, you know,taking, you know,
38:05very long-term decisions and thinking,you know,abstractly.But yeah,I think,you know,ultimately the goal was the same.It's to like get,
38:14you know,the right behavior out of these systems.And that was very interestingbecause the example that you just gavewas saying,you know,a lot of the fairness problemsthat we're dealing with right now
38:23are actually value alignment problems.Yeah.Which is like the problem there is basicallythe system doesn't behave in a waythat's like consistent with,human values.Yeah, yeah.So that's a fairness case.
38:33And then, you know, so like, you know, that's the F in the fat acronym.I mean, to take accountability and transparency, I think there's also common cause.So, you know, one of the issues I've been toying with recently is that that transparency mightbe a way of avoiding certain, you know, international conflicts or it might be part of the toolbox.So historically in arms control agreements, like around nuclear weapons and chemical weapons,there have been things like on-site inspections and, you know, satellite.light monitoring and all these tools that are sort of bespoke for the purpose of the domain.But the general concept is we would be better off cooperating and we will verify that that
39:12behavior is actually happening. And so that, you know, if we detect defection by the SovietUnion or the Soviet Union detects defection from us, then they can respond appropriately.But, you know, we can build, you know, trust but verify in Reagan's terminology. And I think ifif you actually had the full development of the fat methods and you had accountability and transparencyfor even general AI systems or super intelligent systems, I think that would open up the door for a lotmore collaboration. If you could sort of credibly commit to saying, okay, you know, we're developingthis general AI system, but, you know, these are its goals or this is how it learns its goals.And, you know, we're sort of, you know, putting these hard constraints on the system such that
39:54it's not going to attack your country or whatever. Yeah. I think what's, I mean, one of the thingsit's so intriguing about it though is like the reason why like fat a g i for me is like oh it's like kind ofkind of a crazy idea is because i know typically in like the literature around aGI it's very much likethe idea that it would be accountable and that it could be transparent is usually consideredimpossible right because like a g i so complex and so powerful that it would like that nothingcould do that but almost yeah i mean the movie you're making the movie you're making is to say likeactually we might we might be able to do it right well there are differences of opinion on like howsort of interactive the development of, you know, an AGI would be and, you know, the extent to which humans
40:33will be in the loop, you know, over the long run. And so I mean, Paul Cristiano at OpenAI, forexample, has a lot of really good blog posts. And, you know, some of these ideas are in the paperconcrete problems in AI safety about, you know, about the idea that, you know, courageability,what he calls corrigibility and what others have called corrugability might actually be like a stablebasin of attraction in the sense that if a system, you know, is designed in such a way thatit's able to like take critical feedback and it's able to say okay yeah what i was doing was wrongthat might sort of like stabilize in a way that it's like continuously asking for human feedback so it'spossible that accountability is you know an easier problem even for very powerful systems than we
41:11realize like you know there are powerful uh you know maybe trump aside there are powerful people in theworld who actually seek out critical feedback and like are aware uh and like want to hear diverseinputs and like want to make sure that they're doing the right thing right but this is actually reallyinteresting because it's like it's both short term and long term again right which is like if we couldget the research community to have certain norms around ensuring that like we are seeking to buildcorrugible systems yeah that that might set the precedent that the iGI that eventually arriveswill be one which is actually consistent with that yeah right versus like not right we actuallyhave control over the design of the eventual thing right i've always had such trouble understanding like
41:51the people who thought there are these AI engineers that were trying to take over the world with their AGI.It's like, no, they're going to die too.Like all the incentives are aligned.You just like imagine this apocalyptic scenario.But do you guys have, you have strong opinions on people working in public versus working in private?I know there's like somewhat of a debate around development.Yeah, so you mean like working the U.S. government versus.No, no.
42:15Sorry.Do you have an opinion on like trying to build an AGI in holding some amount of your data?or training data, like publicly versus private data.Yeah, so that's a super interesting question.And I think, you know, we sort of broach the topic in this report on the malicious uses ofAI because I think there might be specific domains in which, you know, maybe it's not,maybe in a world in which, you know, isn't necessarily the world we're in today,but maybe in a world in which, you know, there are millions of driverless cars.
42:42And, you know, they're all using the same, like, convolutional neural net that is, like,vulnerable to this, like, new adversarial example that you just came up with.you might want to, like, give those companies a heads up before you just, like, post out an archive and then someone can, like, cause tens of thousands of car crashes or whatever.So I think, you know, we might want to think about norms around openness in those specific domains where, you know, the idea isn't to, like, never publish, but it's to, like, have some sort of process.But, yeah, as far as general AI and research right now, the community is pretty open.And I think it's sort of both in the, you know, broad interest and in the individual interest of companies to be fairly open because they want to.to recruit researchers and researchers want to publish.So I think, yeah, there's a pretty strong norm around openness, but if we were in a world
43:27where there was, like, more widely perceived, you know, great power competition betweencountries or where the safety issues were a lot more salient or there were some, like,catastrophic misuses of AI in the cyber arena, then I think people might think twice.And it might be appropriate to think twice if, you know, your concern is that, you know,that the first people to, you know, press the button, if they're not, you know,conscious of all the safety issues could cause a huge problem.Yeah, I'm very pro open publishing.Like, I think, like, it should be the default.
43:57And it's like, I'm still disputing situations where I'm like, you shouldn't publishon this stuff.Just because, like, I think it is actually to the benefit of everybody to know what thecurrent state of the field is, because it allows us to make, like, a realistic assessment,regardless of whether or not you believe in AGI or you believe in superintelligence,like, you know, like, it's useful just to know, like, what can be done.because even if you're thinking about the more prosaicbad actor uses, right?
44:22Like, it's useful to know, like, what are the risks?And we can't do that in an environmentwhere, like, lots of people are kind of holding back.And so it's important to know the state of the fieldat any given time so we can actually make realistic public policy.Otherwise, we're really operating in the dark.Yeah, that's a great point.Okay, so, Miles, last year you wrote about predictions for 2017
44:41or 2018?Yeah.Yeah, I made the predictions early 2017,and then I reviewed them like a month ago.Okay.This year, 2018.You can get a full year.I was not prepared for this.
44:52You can have a three-year gap, a three-year timeframe then.Even more.Three years.Sure.Yeah, I think there will be superhuman StarCraft and Dota, too, probably in that time horizon.I said in, I think, early 2017, that it would be the end of, that I gave like 50% chance by the end of 2018.So this gives me more runway.I'll say, you know, they're like 70% confident that, you know, that, you know, they'll be superhuman and StarCard.
45:23I'm actually less familiar with Dota, too.So I'll say just StarCraft.All right.Okay.Tim?I think meta-learning will improve significantly.So this is basically treating machine learning, designing machine learning architectures as if they were their own machine learning problem.It's something that basically is done by like machine learning specialists right now.
45:42And the question is how far will machine learning researchers go in replacing themselves, essentially?and I think that will get really good in ways that we don't expect.And your insight into why that will happen is what?There's some of the results that we're seeing from the research right now.It just seems like these networks are able to kind of tune their parameters in a way that, at least I would have not expected.And so it's cool seeing that adapt and advance.These are all positive things.All right, guys, well, thanks for your time.
46:10Cool.Thanks for having us.All right.Thanks for listening.So as always, you can find the transcript and the video.at blog.w.Ycombinator.com.And if you have a second,it would be awesome to give us a rating and review
46:21wherever you find your podcast.See you next time.
There aren't comments yet for this episode. Click any sentence in the transcript to leave one.