Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas - 220 | Lara Buchak on Risk and Rationality

Episode Date: December 12, 2022

Life is rich with moments of uncertainty, where we're not exactly sure what's going to happen next. We often find ourselves in situations where we have to choose between different kinds of uncertainty; maybe one option is very likely to have a "pretty good" outcome, while another has some probability for "great" and some for "truly awful." In such circumstances, what's the rational way to choose? Is it rational to go to great lengths to avoid choices where the worst outcome is very bad? Lara Buchak argues that it is, thereby expanding and generalizing the usual rules of rational choice in conditions of risk. Support Mindscape on Patreon. Lara Buchak received a Ph.D. in philosophy from Princeton University. She is currently a professor of philosophy at Princeton. Her research interests include decision theory, social choice theory, epistemology, ethics, and the philosophy of religion. She was the inaugural winner of the Alvin Plantinga Prize of the American Philosophical Association. Her book Risk and Rationality proposes a new way of dealing with risk in rational-choice theory. Web site Princeton web page PhilPeople profile Google Scholar publications

Hello, everyone. Welcome to the Mindscape Podcast. I'm your host, Sean Carroll. A couple years ago, a little more than a couple years ago, episode 86 of Mindscape, we had Lord Martin Reese on the show. Martin is a very well-known and respected theoretical astrophysicist, but he's also someone who worries about the future of humanity. So we were talking about existential risks, the prospects of post-humanity, stuff like that. And naturally, we talked about one example that was in the news not long before. which was the large Hadron Collider, the giant particle accelerator in Geneva, which we don't remember now because it didn't happen, but people were worried at the time about the chance that you would make a tiny black hole or a new kind of particle that would utterly destroy the earth. And physicists, of course, worried about this, too. They don't want to die because they did some particle physics experiment

gone terribly wrong, so they thought about it. They thought about it really hard, and they said, as far as we can tell, the chances of this happening are just incredibly, incredibly tiny, we should just go ahead. There were people out there who said, what do you mean incredibly tiny? How tiny does it have to be if we're really talking about the end of all life on Earth? And Martin Rees had an interesting rejoinder to that. He said, well, there's also a probability that you will find a free energy source that will completely remove. our dependence on foreign oil forever.

This is very, very unlikely to happen at a particle accelerator like the LHC, but it's possible. And his point was, you know, once you're going into these tales of distributions where things are super, super unlikely, it's hard to say what probabilities are and hard to judge them. His judgment was you should just go on with something like the LHC. I thought that was an interesting argument, but it does beg some questions about how we deal about huge, terrible risks, or even how we deal about minor risks when we're facing a choice between basically a safe bet versus something that might be okay, but has a chance of being really good or a chance of being terribly bad, how do we reason about that choice

rationally? How do we weigh the possible risk of things going terribly wrong? So today's guest, Laura Butchak is a philosopher at Princeton University, and she is the expert in the world on exactly this problem. She has a book called Risk and Rationality, where she basically addresses some lingering paradoxes in the theory of rational choices. In the theory of rational choices, it's all about, I mean, of course, if you have something that you value at 100 and something else you value at 50, then you're going to prefer the one you prefer, you're going to prefer the 100 over the 50. But when probabilities become involved, how do you risk, or how do you weigh, rather, 50% chance of something you value 100, and 50% chance of getting zero versus 100% chance of getting something you value at 50? And of course, there's a lot of discussion in the rational choice literature about what it means to value something. It's not just money. You certainly value the second million dollars less than the first million dollars that you might ever make.

That's understood. That's okay. But there's a general feeling that there is some way to assign value to things. And then when you're faced with a probability, you just take the expectation value. You take the average. You multiply the reward times the probability, and that's how you should weigh it. And Laura's point is that, no, you shouldn't. There is a separate thing to keep in mind, which is how risk averse you are. So her point is that it's not irrational to either be, super, super averse to taking risks or to not care that much. It's not something that you can subsume into a rational choice calculation. It's a separate thing to keep in mind. People have values for different things that can happen. They also have an aversion to the worst possible thing happening. That is a risk aversion factor that it's okay to introduce into your theory about how to reason in cases of uncertainty. We're all faced with cases of uncertainty all the time. The LHC is an example, but where do you go for dinner? What job do you take? Who do you go on a date with? These are all things where you're weighing probabilities because you don't know exactly what's going to happen. So this kind of

reasoning, this is philosophy that really helps you in the real world in principle. So you'll see what some of the answers are, and you may disagree, but that's the risk you. you'll have to take. Let's go. Lara Bouchach, welcome to the Mindscape podcast. Thank you. Thanks for having me. I have to ask, you know, when you tell people that your professional expertise is in rationality, does this put like a lot of pressure on you to never do something silly, like, you know, forget your keys or things like that? Do you feel like you have to be the most rational person at all times? Yeah, exactly. And in fact, when I tell people my specialty is decision theory, people who know me laugh because I'm terrible at making quick decisions. And in fact, there's this joke that people study the thing they're bad

at. So, you know, other people knew how to make decisions so they didn't have to study it. I had to get a PhD. And I still don't know. Yeah, I mean, famously, moral and ethical philosophers are not always the most moral and ethical people, right? We hope, but there's no real correlation there. That's right. Yeah. So what do we mean by rationality? Like I think that there's a technical sense that you probably have in mind that might be a little bit different than the person on the street has in mind when you say something like that. Yeah, good. So there's two things, two different senses of the word rationality, even in philosophy.

And one is just purely consistency. So you can think of it by analogy to logical consistency. If you believe P and you believe P implies Q, but you don't believe Q, you're irrational purely in the sense of being inconsistent, regardless of what the content of P and Q is. So that's the kind of rationality decision theorist study. The other kind of rationality, we might call substantive rationality, and an example of that is, you know, if you believe that aliens have invaded your body,

that might be perfectly consistent, but we think that's not a good. good belief to have. Okay. Doesn't match the evidence or something like that. Doesn't match the evidence or something like that. That last one is, on the one hand, sounds more, I don't want to say more important, but at least closer to the average person's thought about what rationality is, but at the same time, maybe a little bit loosey or goosey?

I mean, do we have quite as rigorous a definition of what that means? We certainly don't have a rigorous formal definition. Yeah. Of what that means. So are you studying mostly the first part, the nice logical, philosophical part? Yep, studying mostly the first part. And it is, you know, maybe it seems a little bit far removed from what we initially care about. But if you sort of think about consistency and decision making, the big question is if you already know what you want and you know how likely various things are to happen, then what should you do? And that's really a question of being consistent with your beliefs. and desires. So in that one, described that way, I hope it sounds like something that most people are in fact interested in. They want to know, I like these things. What should I do about that?

Rather than which things to like. I can't help tell you which things to like, unfortunately. It almost sounds like something where if you don't think about it too hard, you might get the idea that no one should disagree about this. I mean, is there a minority school of thought saying we should be inconsistent and irrational all the time? Or... Maybe a very small minority, but we don't actually know who they are. No, in fact, everybody thinks we should be consistent, but the real debate is about what consistency means. Right. Okay. And you also mentioned decision theory, which is that the same? Is that a subset of being rational? Or is that like a neighbor? Yeah, so we can have rationality in a number of different areas. There could be rationality in what you believe, rationality in what you desire.

decision theory is about rationality in what you choose or what you prefer. And really, it's about rationality in what you choose or prefer when you're in a condition of risk or uncertainty. So when you don't know what the world is like. Which is most of the time. Which is most of the time, unfortunately. Okay. So that is the tricky part. And I've talked to people like Julia Gallif on the podcast, the people who are sort of, you know, professional.

rationality boosters, but not within academia quite. And they talk a lot about Bayesian theorizing and updating on evidence and things like that. Does that come in also to either the rationality part or the decision theory part? Yep. Yeah. So one part of decision theory. So decision theory involves at the very least probabilities, which measure how likely you think various things are to happen and utilities, which roughly measure how much you like various things. And so the Bayesian updating that all falls into the probabilities side. So one thing we have to know before we make a decision is what should my probabilities be? And especially given the evidence and how should my probabilities respond to evidence.

So that's part of decision theory. That's not the part where I've argued for something that's different from the mainstream. And I guess, you know, still laying the groundwork a little bit here, how concerned as a philosopher like yourself with the reality of human psychology. Like, are you trying to describe how rational people actually behave? Or are you content to just think, like, this is how you should behave if you were rational? I don't know how people actually behave. Well, I'm trying to describe how people should behave.

But how people actually behave is relevant because if we've sort of, if people behave very differently from the theory, and we don't have a good explanation for that, then we're going to want to investigate whether the theory is actually correct in that place. That sounds remarkably like science rather than philosophy. Yeah, well, they're maybe coextensive. I'm happy to believe that myself. So, yeah, so in particular, we might imagine that the armchair thinking about rationality leads us to a certain picture, and then we notice people not behaving that way, and then we go,

oh, actually we made a mistake, we can be better at describing what it means to be rational? Yeah, or at least we need to, but how people behave isn't relevant, isn't directly relevant to what rationality is. It's just sort of like a datum and we either have to say, oh, that's a way in which people behave irrationally and here's why that's irrational or we have to incorporate it into our theory. So we won't know which thing to do until we investigate further. Is there any guidance other than consistency to what we would eventually dub to be rational?

Not in decision theory, but consistency turns out to be a pretty difficult notion to say what it means, and it actually turns out to have a lot of content. So consistency really dictates what you should do pretty strongly. Okay, that's very interesting. All right, so let's get into the meat of it then a little bit. It might seem, once again, if you were very naive, you would say, what's so hard? I prefer A to B. I'm going to do A. That's my decision theory. But where it becomes interesting is when uncertainties come into the game, as you already mentioned. Exactly. So how does that happen? How do we think in conditions of uncertainty? Well, so I like A more than B and B more than C, and I have the following choice. Do I want to just get B for sure? Or do I want to have some chance of A, but some chance of C, which of those two things is better?

And just knowing how much I like A, B, and C isn't going to tell me which of those two choices I should make. So what do we do? So what do we do? Well, first of all, we have to know or estimate the probabilities of A and C. So, for example, maybe I'm, let's put some numbers on this. Maybe I'm, I like $100 more than I like $50, more than I like $0. So I want to decide between just getting $50 or taking a coin flip of a fair coin between $100 and $0. So yeah.

And so what should I do? Well, the first step is to figure out what decision theorists call my utilities. That's how much I like the various amounts of money. The way that we typically do that is to. reverse engineer them from your preferences. So I know lots of your preferences over a lot of different things. From those preferences, maybe I discover that your utility function is linear in money. So you think $50 is better than $0 by as much as you think $100 is better than $50. Now, according to the standard theory, the one that I argue against in my book, that's actually all the information we need.

Once we have your utilities and your probabilities, we can figure out exactly what to do. So on the standard theory, you should maximize expected utility or average utility. So the value of a gamble that has a 50% chance of giving you 100 utility and a 50% chance of giving you zero utility is just going to be 50. Multiply 0.5 by 100, multiply 0.5 by 0, add them together. And the utility of just a straightforward $50 is going to be 50. So according to expected utility theory, you have to be indifferent between those two choices. at least given that your utility function is linear. Now, if you're not indifferent, the expected utility theorist is going to say, well, then your utility

function isn't what we thought it was. Right. And so in particular, many people would have a utility function that has less utility for the second $50 than for the first. That's right. That's right. So if you're, if you prefer the $50, the share. sure thing, $50 as most people do, then the explanation, according to the expected utility

theorist, has to be exactly what you said, that I like the first $50 more than I like the second $50, which means, on average, that coin flip will have less utility. But the idea is that even if that's true, even if you'd rather have $50 than a 50-50 chance of zero or $100, that's no worries for expected utility theory because it's just a $1,000. that utility is not the same as money, and we can still have something called utility. That's right. That's right. Is that true? Like, we're not even getting into your stuff yet, but I've always felt a little bit nervous at the idea that there's something called utility that is a number that we can just attach to different outcomes.

I know that maybe we can reverse engineer such a thing, but is that an innocent move or is that fraught with peril? Yeah, great question. Well, there are maybe two moves there, and one is the move that we can have a number that we reverse engineer from your preferences, that describes how much you like something that's different from your money. The second move that people often make is something like, and utility doesn't describe anything in the head or in the world. It's simply a representation of your preferences. So it's meaningless in itself. So I tend to think the first move is innocuous. It describes something pretty intuitive, right?

So like how much I like money depends on how much money I already have. That's like a pretty good. It's pretty good. We want to be able to capture that with our theory of decision making. Maybe which coin flips a millionaire takes is going to be different from which coin flips someone in poverty. takes. So that seems innocuous. The other move and then saying, well, we don't need utility to describe anything in the head or we don't need the numbers to sort of fit intuitively with what you think your utility function might be. Maybe that's a little bit less innocuous.

So that kind of move would say, okay, Sean, you prefer the the $50 to the coin flip, you think you don't like additional dollars any less, the more money you have. But as it turns out, your utility function says otherwise. So you're really bad at figuring out what your utility function is. Right. I mean, maybe you just answered the question I wanted to ask, but let me ask it anyway in case there's another angle here, which is I worry about the circularity of it.

Like, if I already had perfectly rational preferences, then I could imagine I could reverse engineer or utility function to represent them. But in some sense, you're defining what it means to have perfectly rational preferences by I'm always going for the thing that has greater utility. So if real people don't have perfectly consistent preferences, what do we do? How do we even talk about this? Good, yeah. Good.

I mean, that's a great question. And it's a question that's received a lot of attention. an answer you don't want to give is that unless someone is perfectly rational, they don't have a utility function at all. Okay. And our theory would be pretty useless, right? Some of my best friends are pretty irrational, so yeah. Oh, well. But they still like things, right?

So we want to still say they have a utility function. So I think a better answer is that often many of your preferences, or many of your central preferences are pretty consistent. And you do have maybe good access to your central preferences. And then we can figure out what your more complicated preferences should be from them. For example, maybe if I ask you about a bunch of coin flips between dollar amounts, maybe that's something you have pretty good access to and can tell me about. And then I can figure out your utility function from that.

And then I can then use your utility function when I advise you how to play poker or how to invest in the stock market. Because those are much more complicated decisions that you don't have any intuitive preferences about. So I kind of like to think of it as there's this first step of determining your utility function from a smaller subset of your preferences. Okay. One that you're pretty sure about. And if those are inconsistent, maybe you can. can figure out which ones you want to keep and which ones you want to jettison. And then second step, tell you what to do in these more complicated actual decisions. Okay, that makes sense,

because that's related to what I was going to worry about, like whether or not people even implicitly actually do anything like this. And maybe what you're saying is they do something close to it for the big, important, straightforward decisions. And then that implies a certain way they should be acting rationally in the fringe cases, even if maybe they're not as good at that? Yeah, that's exactly right. Yeah. Don't want to say, I mean, I don't think people have anything as precise as the utility and probability numbers that we use in the theory.

But I still think the theories are useful because they can tell us things in broad brush. So maybe we shouldn't use the theory to distinguish between. how you feel about a coin flip with a 60% chance of heads and a coin flip with a 61% chance of heads. Okay, maybe you don't actually have any feelings about those two things that are different from each other. But if we can sort of, in general, dry out lessons from the theory, like here's a lesson on the classical theory.

If you are an expected utility maximizer, then you're always going to want to make decisions on the basis of more information rather than less. Like that's a pretty simple principle that we have to do this complicated math to discover. But the theory is sort of useful for giving us insights like that, even in cases where the complexity of the theory outruns the actual complexity of our heads. I know that in some circumstances, in some contexts, people will make a lot of a big deal about the fact that human beings are bounded in their rationality. and therefore we satisfy instead of optimize. Satisficing being some weird combination

of satisfying and what is the rest of satisfice made out of? Oh, that's a good question. I forget. I don't know. It's a terrible word anyway, but like we do good enough and then we stop because it would take a lot of... Like satisfy but mathy.

Yeah, exactly. But so is that part of the being a decision theorist or a rationality theorist game that the real people like don't do all this complicated math, but they kind of approximate it in some way? Yeah. So the ultimately decision theory is about what's rational to do or what's rational to prefer, but it doesn't need to be about what's the best method to use to make decisions. So even if we think expected utility maximization is the right theory of rationality, it might be the case that you nonetheless shouldn't use it in a lot of decisions.

For example, a car is coming at you. You need to swerve to the left or the right. If you bothered to calculate the expected utilities, it would take too long and you just get hit by the car. So it's like not a good decision-making method in all circumstances. So there's a separate question of what are the best rules or heuristics that are going to help us approximate ideal rationality. Is it analogous to the issue?

in political theory about ideal theory versus sort of real world stuff. Like we can imagine what the perfect society is and then move toward it, but then other people are saying it's not even worth imagining the ideal society because our minds might change along the way. Yeah, that's one analogy. But decision theorists really are interested in ideal rationality first and foremost. I think maybe a different analogy is the paradox of hedonism. So if you're a utilitarian, you think that, or a hedonist, you think that the best moral actions are the ones that produce the most pleasure in the world.

Right. Nonetheless, that wouldn't be a very good method for making decisions about what to do. You wouldn't say, oh, hello person, let's be friends, because my being friends with you would maximize the amount of pleasure in the world. That would be a method to not make friends and then therefore not have a lot of pleasure in the world. So instead, there's just going to be a separate evaluation of what is good, in this case, what brings about the most pleasure in the world, and how to actually behave or think or live your life to meet that ideal standard. And similarly, in decision theory, there's one question.

What do rational decision makers choose? And another question, how do they go about choosing it? Okay, fair enough. And I guess there's just a couple more minor points to get on the table here before we get into risk, which is the other fun part of what you're thinking about. We use the word utility, but this is personal decision making. It's not necessarily connected to utilitarianism, which imagines that we can sort of add up people's utilities.

Yeah. That's right. So the word utility in decision theory has come to mean just whatever it is you care about. So you can still be maximizing your utility, your own personal utility, even if you're perfectly altruistic. The decision theorist would just say, oh, Sean's utility function assigns higher utility. to things that are good for other people and lower utility to things that are bad for other people. Right. So it's very flexible in that way. I mean, do people ever claim, there's got to be some

philosopher out there who makes every possible claim, right? Are there claims that there are situations where people are acting apparently rationally that simply cannot be modeled by attaching utility function to them? So in order to attach a utility function to someone, we're going to need the utility function to have certain properties. And if it doesn't have those properties, then we can't. So one example might be if you assign infinite utility to something, then that's not going to,

we're not going to at least be able to maximize expected utility or average utility. So that might be an example. It's perfectly rational to care only about. this one thing so that it's infinitely valuable. And yet that can't be captured by expected utility. Although I suppose that's still a case of wanting to maximize utility. Yeah. And also, as soon as you start talking about infinity, I think that probably this is not a

realistic kind of thing. Yeah. I can't place infinite utility on anything at all. But I guess, okay, the other footnote is we talked about how maybe the second $50 is less valuable to you than the first $50. But there's nothing in the formalism that prevents a person from being risk seeking, right? Like they would rather have the 50-50 chance of $100 than a sure thing of 50, right? That's perfectly okay.

Yep, that's right. You can prefer anything you want as long as you're consistent. And is there psychological data about where people usually fall? Well, people tend to be risk-averse when it comes to buying insurance. So you're, you know, say you have a one in a hundred chance of a fire in your house, you'll pay more for that insurance than is one in a hundred the cost of your house. And people tend to be risk seeking when buying lottery tickets. Okay. So if there's a lottery that has like a one in a million chance of winning a million dollars, you might nonetheless pay more than a dollar for that lottery.

I guess that, yeah, that's a great example. I'm kind of ashamed of I didn't think of that. So playing the lottery or many other games in Vegas or whatever are examples where we purposefully take risks, even though the expected payoff in dollar terms, is less than what we're paying for the right to take the risk. That's right. Yeah, that's right. Do we ever have the professional rationality conference in Las Vegas? Has that ever happened? Have we seen whether the philosophers go play the slot machines? There was an APA once, an American Philosophical Association in Las Vegas. And, you know, as a decision theorist, you look at those slot machines. They just look like trash cans for money.

Good. So there is an example where the philosophers are doing well at their particular area of expertise. Okay, so the risky business, as it were, is the important thing here, right? I think you said it once, but let's say it again. If I get it right, in expected utility theory, all you need to know to decide what is the irrational choice is what are the utilities of different outcomes and what are the probabilities of those outcomes happening, right? That's right. And you would like to say that, well, actually, let me first ask, are we any good at really evaluating probabilities? The examples are always like flipping coins and going to Vegas or whatever.

But if it's like real world things, should I take this job or something like that, that sounds hard. Yeah, it's pretty hard. I think some things like we're pretty good at, you know, we get weather forecasts. They're usually pretty specific, 30% chance of rain. We're pretty good in those cases where there are possible experts that can tell us things and usually they're phenomena that behave pretty regularly like, you know, like weather or cards or maybe the stock market doesn't behave super regularly, but a little more regularly than my job prospects behave or something like that.

Okay, so then, but your big argument in the book and elsewhere is that maybe this minimal set of ingredients of knowing what all the options are with their utilities. and the probabilities of them happening is not all that we can or should take into account. And the examples from your book, I know, I had a friend noticed that I was reading your book and picked it up and they're like, this is very technical. Some of the examples get very complicated. But is there a simple way to ease us in to this discussion? Yeah.

Yeah. So here's a simple way to ease us in. Here's something you might care about. you might put more weight on worst-case scenarios than better-case scenarios in the sense that you might whatever happens in worst-case scenarios might play a much larger role in your decision-making than what happens in best-case scenarios. If you do that, then there isn't going to be a utility function and a probability function that can capture your decision-making. So the, So expected utility theory is going to say you're irrational.

Okay. Do you want to talk about some examples? We got to do some examples because I would think when you say that, well, isn't it saying that we worry about the worst possible outcome just to say that we put a big negative utility on the worst possible outcome? So if we did that in one decision, then that's what expected utility could say. But if in every decision you care more about the worst case, than the best case, there isn't going to be a utility function that can capture you. Okay, okay.

Good. So maybe an example would help make that clear. Okay. So there are two examples I like. One is called the Allay paradox. Maybe we could talk about both of them and see if you find either of them compelling. Great. Okay, so may I ask you, Sean, for some of your preferences?

Please. Okay. So here's two gambles. You can have either one. The first gamble, a 10% chance of $5 million. And the second gamble, an 11% chance of $1 million. I'll take the 10% chance of $5 million. I'll take the 10% chance of $5 million.

Okay. Wait, sorry. Yeah, so say it again to make sure. Yeah, yeah. So a 10% chance of $5 million or an 11% chance of a million. Yeah, I'm just going to be the rational person and take the 10% chance of $5 million. Sounds great. Most people share your practice.

Yeah, I'm not weird that way, yeah. Okay, there's a second choice. Forget that first choice. Now, this one is slightly more complicated, so you might have to listen hard. Yeah, write it down. Okay, so an 89% chance of a million dollars and a 10% chance of $5 million and a 1% chance of nothing. So that's like an entire gamble. 1% chance of nothing,

89% chance of a million dollars, 10% chance of $5 million. So probably I'm going to get a dollar, maybe I'll get $5, tiny chance I'll get nothing. Yeah, millions of dollars. These are millions of dollars. Millions of dollars.

Does it matter? Does it do the units matter? Oh, actually, that's an interesting question. It is, yeah, okay. Yeah, we can think about that. Millions of dollars, good. Millions of dollars. Okay, so that's one option.

Second option, a million dollars for sure. Okay. So which of those two would you rather have? That complicated gamble that's probably going to get you a million dollars has a one percent chance of getting you nothing and a somewhat chance of getting you $5 million. Or would you rather me just give you a million dollars? So just to talk it through, I mean, I think the answer is I would like the first one better.

And I think that's probably the answer you want me to give. but you don't care. This is not a normative discussion. Just want to get to know you better, Sean. That's all. This is a weird dating site, asking people about their rational preferences under conditions of risk. So the first option says, again, most likely a million dollars, but a non-negligable chance of $5 million and a tiny chance that I get nothing at all versus a hundred percent chance of a million dollars. And I think that the reasoning would be that I'm willing to risk the 1% chance of nothing to get the 10% chance of 5 million.

Yes. So that's your actual preference? Yeah. You'd rather have that one. I'd rather have the first option with the 89 and one. Okay. Well, most people would rather have the second option. Really?

Mm-hmm. Most people would rather just have a million dollars than to risk getting nothing in exchange for maybe getting $5 million. So just to make it, just to, you know, I'm going to keep saying it out loud so everyone follows along. So in the mixed probability one, there is a 99% chance that I get at least a million dollars. And within that, there's a sum chance that's five million. And I'm risking the 1% chance of getting nothing. And I do know, as someone who's played poker, that 1% chances do happen. So I'm not, my reasoning is not that 1% is equal 0, because that's certainly false.

But I think I'd be willing to take that much risk. Okay. Yeah. Okay. Well, that's interesting. Most people are unlike you. Okay. In that respect.

And so your preferences actually can be captured by expected utility maximization. Okay. But if you have the standard preference, here's sort of how the reasoning goes. If I'm choosing between a 10% chance of $5 million and an 11% chance of $1 million, that first choice I gave you, I will I'll probably get nothing The minimum value of that is nothing So close

Whether I'll get 5 million or 1 million So go for the bigger option But in the second choice I gave you One of the options Gave me a million dollars no matter what The worst I can do in that gamble Is get a million dollars

And so if you're the kind of person That weighs worst cases heavily, you don't even have to weigh them that heavily. You just have to weigh them sort of more than you weigh best cases. Then your expected utility maximization isn't going to be able to capture you because there's no utility function that allows for that pattern of preference. Good. So maybe said in another way, there is a rational way of behaving under which the introduction of even a tiny chance of a super bad outcome weighs very, very heavily in your decision process. Yeah. Yep. That's right. That's right. And by the way, you mentioned poker. And I just

want to point out that your reasoning in poker might be a little, maybe it should be a little different in this case than in poker. Because when I play poker, I'm going to be playing a lot of hands. So whatever losses I make are going to be made up by gains on if I just pick the thing that does best on average. On the other hand, if I'm just going to make a choice once, like these gambols, I'm only going to offer them to you once, say, or the choice of which job to go for, the choice of who to marry. We can't employ the reasoning that says, look, if this were to be repeated many times. I'd do better on average by taking this thing because it's not going to be repeated many times. I think it's a very, very good point. And even within poker,

there's a difference between playing just at a table versus playing in a tournament, right? Because playing at a table for exactly the reasons you just said, you in every hand want to maximize your expected return. But that strategy might not maximize the chance that you win a tournament, because if you bust out, you can't win, right? You can't climb back in. Yeah, yeah. And I also, I appreciate the idea that if I were to be given these two options and I got unlucky and I had to come home and tell my spouse that I just won $0,000, even though I could have won a million dollars, that would be a difficult conversation to have. That would be a difficult conversation to have, yeah. Which is why this is a really good dating question. Right, exactly. Make sure you're compatible

about these kinds of things. Exactly. Okay. Can I give you one more example? Please, yeah. Okay. So, and the LA paradox, that's the example I just mentioned, that involved large amounts of money and small probabilities. So we might think maybe people are getting confused by the big numbers or something. So here's an example that doesn't involve large amount of, doesn't involve values like that.

And this is an example due to Matthew Rabin. So let me just ask you, I'm just going to give you a coin flip. between losing $100 and gaining $110. Okay. Do you want that coin flip or not? No. No. Okay.

Most people don't want that coin flip. Now let me offer you a coin flip between losing $1,000 and gaining $6 million. I'll take that one. You'll take that one. Good. Most people would. In fact, this is another example. example of a preference that expected utility can't handle.

And the basic idea, yeah, the basic idea is that if you don't like that initial coin flip between losing $100 and gaining $110, then if you're an expected utility maximizer, your utility function has to be pretty concave. It has to say you like $110 less than losing, than the, you know, you like the interval between 110 and zero. That matters less than the interval between zero and negative 100. And if your utility function is already that concave in that low amounts of money, then once you get to losing $1,000, that's actually going to be more valuable than gaining any amount of money whatsoever. So the point is, if we start in order to explain these a small amount of risk aversion in low numbers,

we have to imply an implausibly large amount of risk aversion in large numbers. So this sounds like a result that is a little mathy and maybe you're hiding some of the math from us, but we'll trust you that is a result. But the, I mean, I bet I could wriggle out of it if you gave me arbitrary freedom in my utility function. And presumably by utility function, you mean assigning different amounts of utility to these gains and losses and money. The only restriction on your utility function is that it has to, it has to be, well, using a math, math term everywhere differentiable. It's a piece. It has to be smooth.

Really? That's all? Yep. Yep. Yep. Yeah. And the sort of like intuitive idea is that if you, if you don't like this initial coin flip, it has just a little bit of gain, a little bit more gain than loss, then you really don't value positive amounts of money very much. But I'm surprised that I'm not allowed to not value $110 very much, but value $6 million a lot. Right.

Well, so, oh, sorry, I guess the other, the assumption is that your utility function is going to be concave. So if you had a utility function that was, you know, increased very slightly up to 100, $1,000 and then like shot in the air, then you'd be allowed to do that. Okay, good. All right. But we might think if we sort of asked you. And the actual result is that you don't like this coin flip regardless of your initial wealth level.

Then you don't like the losing $1,000 gaining any amount of money whatsoever at your current level. So this seems to be even more vividly than the first one. As you said, a real challenge for expected utility theory. So just to say it once again, given the choice between losing 100 and gaining 110 on a coin flip, most people will not want the coin flip at all because it's barely a difference and you don't want to lose $100. Given the coin flip between losing $1,000 and gaining $6 million, most people would willingly take the risk of losing. a thousand to gain the six million at a 50% chance but there is no way to accommodate that within expected utility theory under certain reasonable assumptions yeah that's right

and can I give you one more example just because it's all a lot of them are pretty mathematical this is an example that is just supposed to illustrate that there are two different psychological phenomenon okay so maybe we want to capture them in our theory of rationality Okay, so you really like coffee. In fact, if you sort of think about it, you think I like the second cup just as much as the first. I like the third cup just as much as the second, the fourth cup just as much as the third. You're a coffee addict. I, on the other hand, can't tolerate that much coffee.

Once I've had one cup, I don't want another cup. Now, you and I, so you and I have different attitudes towards coffee, but we also have different attitudes towards gambles. So for you, you think the possibility of not doing so well is not going to make up for the possibility of doing pretty well. So you weigh worst case scenarios really heavily. Me, on the other hand, I weigh all scenarios the same. So let's think about each of our preferences between one cup of coffee on the one hand and a coin flip between no cups of coffee and two cups of coffee. Well, I don't want any more than one cup. So I'm going to take one cup of coffee.

Guaranteed, yeah. You like each cup just as much as the last, but you don't like taking risks. You weigh the possibility of not getting anything more heavily than the possibility of getting the two cups. So you also prefer one cup of coffee. Now, because we have the same preferences, expected utility theory is going to have to say we have the same utility functions. But there are two really different psychological explanations for why we chose how we did. Why we chose how we did. And you might think we'd like a theory that captures the differences between our two preferences.

I would like that theory. So what is that theory? That theory is instead of maximizing the average utility, you maximize a weighted average where if you give more weight to the bottom portion of your probability distribution, then the top portion, I call you risk avoidant. And if you give more weight to what goes on at the top portion, I call you risk avoidant. I call you risk inclined. So in other words, in addition to just having utilities for different outcomes and probabilities of getting them,

we also each have a risk aversion curve. That's exactly right. And it's convex if you're risk averse. So as you're thinking about what happens in worse and worse cases, you care about it more and more. and it's concave in the case of risk inclination. So as you're thinking about what happens in better and better scenarios, you care about them proportionally more and more. Okay.

And you don't care. They're both very, you can fit them both into the theory. That's exactly right. And I think both people are behaving fully rationally. Right. So that's the big thing. So there might be other people, philosophers who will remain nameless, I guess, who will say, no, these people who aren't following what we think are their utility functions are

just irrational. And you're saying that, no, they just have different attitudes towards risk, but under the idea that being rational is just being consistent in your preferences, they're still perfectly rational. That's exactly right. Okay, good. And so what is the, I mean, maybe put some more meat on these bones. Can you apply that idea to the examples or some of the examples that we're doing?

Sure, sure, sure. So in the example of the first example with the million dollars and the $5 million, you or the standard person, not you, because you didn't have the standard preferences, but the standard person assigns more weight to the bottom 1% of the distribution than the top 1% of the distribution. So the fact that you might get $0 is that gets weight maybe 0.2 instead of 0.01. In the example with the coin flips, maybe most people would assign weight three quarters to the bottom to the worst option. and one quarter to the better option, given that they both have a 50% chance of happening. So even if you had a linear utility function, if you're assigning wait three quarters to losing $100 and wait one quarter to gaining $110, that's going to work out to be a negative number. So less than zero, i.e. doing nothing. Okay, good.

And then in a coffee cup example, your utility function of coffee cups was maybe utility of X coffee cups equals X. My utility function of coffee cups was maybe utility of zero coffee cups equals zero, utility of one or more coffee cups equals one. My risk function was linear. So all states get weight equal to their probabilities, whereas your risk function assigned, you know, maybe two-thirds weight to the bottom 50% of the distribution. Okay, I think I am seeing it. Let me modify the example of the $1,000, $5 million, nothing. Because I think it – maybe this is not a realistic choice, but I think it shows – It helps me understand why a typical expected utility person would get in trouble with this.

I could easily imagine if you said there's a coin flip and you get either nothing or, let's say, $20 million. Coins flip between nothing and $20 million or guaranteed $1 million. I think I would take the $1 million guaranteed because I am that risk averse even though. It's not even about risk aversion in that case. it's about my utility function, right? Like the $20 million is not as valuable. But if you said to me, 98% chance of a million, 1% chance of nothing and 1% chance of 20 million,

I might prefer that to a guaranteed million. Because my thought process would be like, you know, probably I'm getting a million, but I kind of like the idea of having a little bit of a chance of getting $20 million. And so basically it's exactly, exactly the same comparison, but with an extra 98% chance of a million dollars guaranteed. And so if I do prefer one example, then I should be consistent in my preferences according

to expected utility. And I'm not allowed according to expected utility theory to just accept the chance of a loss because I like the thrilling chance of a 1% huge gain, but not have the same thing when it's 50-50. That's exactly right. Yeah, that's exactly right. And the preference you just described yourself as having is a risk-inclined preference, by the way. So you care more about what happens at the top 1% of the distribution than at the bottom 1% of the distribution.

But that's compatible with me if I were just given 50% chance of nothing, 50% chance of 20 million, not preferring that to 100% chance of 1 million. That's right. On my view. On your view. Okay. Yeah, yeah, good. Yeah, that's right. So I guess as a physicist, I would worry that there's a curve fitting going on here.

You've added a whole new parameter to our model of rationalities. You're able to fit more circumstances. But, you know, there's a famous line about how if you give me five parameters, I can fit an elephant or something like that. I mean, is there a worry that we're just giving people too much freedom? Or is that just the real world? We're just being realistic. Well, this is why for me it's important to really try to figure out what we're capturing with these parameters. So yeah, you're absolutely right.

If you gave me more parameters, I capture more preference, it's great. But what I'm interested in is whether these parameters sort of capture what it is to reason instrumentally, which means reason about taking the means to your ends. And I think that choice involves not just taking an average. That's one way you might reason. You might take the means to your ends. But importantly, it's actually open to you what to do. It's open to you how much weight to put on worse cases or better cases.

Yeah. So it's in that sense. data fitting isn't the ultimate standard here, including because it might be that people are irrational. So this is a theory of rationality. It's probably not going to fit the data. So our data here, so to speak, is more philosophers data about what we should and shouldn't feel allowed to prefer and what could count as reasoning or preferences that are taking the means to your ends. Okay.

And maybe like another way to think about it. If it's maybe hard to conceptualize probabilities, you can instead think about each possible state of the world as representing one of your future possible selves. So like there are 100 future possible shans. What would you rather giving all the future possible shans a million dollars or giving 98 of them a million dollars and giving one of them nothing and one of them 20 million dollars? And whereas the expected utility theorist will say there's a unique answer to that question about how people, how Sean should value his future possible Sean's. You should give them all equal weight in decision-making. I say, no, actually, it's up to you.

If you want to put more weight on how things go for worst-off possible Sean, that's a reasonable way to take the means to your ends. That's a reasonable way to sort of like cash out the maximum of, I'm trying to get what I want. on the other hand if you as i guess you do put a lot of weight on best off future possible sean that's also a reasonable thing to do in either case you know you only have one life to live only one of these guys is going to be actual sean so it's up to you to um yeah to think about how much weight to put on each of their interests knowing that only one of them will be actual

You know, it only now dawns on me. This is very embarrassing, but I have to think about this in the context of the many worlds interpretation of quantum mechanics, which I'm kind of a proponent of. So the whole point of many worlds is that what we think of as probabilities really are actualities. Well, quantum probabilities, not every old probabilities. But if we did our choice making via some quantum random number generator, then yeah, I've always taken the line, this might be a life-changing moment for me, because I've always taken the line that it doesn't, there's no difference in how we think ethically or morally in many worlds versus just a truly stochastic single world. But you're saying, well, my attitude towards risk aversion could in principle change that. Yeah, that might be right. Yeah. So if you think, if you think all possible shans are actual shans, that's the view, right? then actually you're facing the problem of distributed justice. So you should decide as you might decide or plausible that you should decide as you might decide

when you're just thinking about how much to weigh the interests of each of 100 people. And we have a similar question in that space. Should I care about the average person, how things go on average? or should I give more weight to the people that are worst off in virtue of the fact that they're the worst off? And if you think the latter thing and you have the interpretation, physics interpretation, you just explained, then maybe you should be risk avoidant. Oh, all right. I want to have to think about this. We got to sleep on this before I actually have an epiphany here. But I was just actually, I don't know if you're on Twitter, but I was just actually arguing with David Papineau on Twitter about exactly this. Because he was trying to say that there are moral implications.

of many worlds. And I was trying to say, no, there's not. So I might have to back now. Well, there is one, and I mentioned, there is one difference between the case that maybe if you don't want to rethink your entire life, you might be happy to hear about. So in the case of you distributing over all the possible shans, they're all you. So you might think that you're allowed to follow a different ethical principle when they're

all you. than you are when they're all strangers. Yeah, but I think that to be consistent in many worlds, they're all future versions of me, but once they come into existence, they're all separate people. I see, okay, I see. Yeah, they have to be treated separately.

You reject the idea that I can just treat different. Okay, well, then, yeah, you might just have to up the creek. Up to Creek. While I think, well, all of the words that you have been saying right now, I mean, it's clear there's a relationship here between a slightly different context, which is some kind of Rawlsian theory of justice, right? So John Rawls famously imagines that we try to put ourselves in an original position where we don't know who we are in the world and ask, how should we arrange the world in the way that we'd be happiest with?

And he ends up with a very, what might be called, a very risk-averse suggestion that we want to arrange things to the benefit of the worst-off people. So is that a real connection or is that just a rhetorical similarity? That is a real connection. So there are actually two people writing in this area who ended up with different views. And one is John Rawls and one is John Harsani. So they both agree that when we're thinking about what's best for society, for example, take a simple choice. should everybody have $50 or should half our people have nothing and half our people have $100? They both thought making that decision is the same as making the decision which society would you rather live in if you didn't know ahead of time who you were going to be.

But they had really different views about how you're, would make the decision. So Harsani thought, well, you assign equal probability to being each person and you maximize expected utility. Therefore, both the societies are equally good. You're indifferent between being an arbitrary member of each society, therefore each society is just as good as the other. Rawls, on the other hand, thought, I can't assign probabilities to my being any particular person in this society. And when I can't assign probabilities, I use a maximally risk-averse or pessimistic or ambiguity-averse rule. Namely, I pick the society or I pick the choice with the highest minimum. I maxing in. Therefore, he would rather live in the society. He would rather be an

arbitrary member of the society in which everyone got $50. So he thinks that's the moral choice. Now, if you'll notice, that's a really, really risk-averse view. That's one that turns only about the worst-off person, right? The second worst-off person, we don't care about how things go for him at all. In terms of our society, that might be like caring only about the poor, not giving any weight whatsoever to the middle class. That doesn't seem good. That seems like way too concerned with one part of the same. the distribution. And Harsani, on the other hand, who derives that we should care about the average

member of society, that's not sensitive to how our utility is distributed as all. You might think that's really bad. It would mean a society with really poorly off people and really well off people is just as good as a society in which everybody's sort of in the middle. My view allows for an intermediate position between the two guys. Because I think in the original position, if I don't know which member of society I'm going to be, I should use a risk avoidant decision theory. So I should give more weight to worst case scenarios, but not exclusive weight to worst case scenarios. So maybe if there are just three people in my society, the worst guy, the middle guy, and the best off guy, I should give, I don't know, weight,

wait two thirds to the worst off guy, weight just slightly less than one third to the middle off guy, and just a little bit of weight to the best off guy. This is going to have the result that I care a lot about how things go for the worst off, but still about how things go for the second worst off. So like, should care a lot about the poor, but should also care significantly about the middle class. And even we should care a little bit about the very rich, though much less than we should care about the other classes. So I think this is a really good result. I think it fits with how many of us, in fact, think about the morality of distributing resources across society.

And I think it allows us to give an answer that's sort of sensitive both to the total amount. of wealth in a society, but also to how it's distributed. And kind of compatible with the idea that at the personal utility level, I would rank $1 billion more than 50-50 chance of $2 billion versus zero, right? I'd rather, you know, that second billion doesn't count as much to me. And similarly, you're saying the happiness of the billionaires in society counts less to you than the happiness of the less well-law of people. That's right.

But they both count. Yeah, that's right. And it's not just that they count less because money can buy them less things. It's that even taking into account how much well-being each bit of money can give you, we should care more about the worst off. And one worry about-Ralls' theory is that it's literally the single worst-off person who is running things. So even if you have a hundred million very happy middle class people, if there's one person who's really suffering, that's no good in his theory. And you're happy.

You're much more willing to put up with that in some sense. I'm willing to put up. You mean, my theory doesn't have the result. That's right. I think you're saying, yes, I'm willing to put up with someone, some one person being a lot more poorly off. if it gets a sufficient number of people a sufficiently amount better off. I mean, on the one hand, has anything like Rawls's criterion ever come close to being implemented in any world? Like, have we ever had a society that really tried to maximize things for the worst off person anyway?

No, pretty much not. But, okay, but again, you've introduced sort of a new degree of freedom, right? I mean, so you're saying, yes, I want to have a distribution that is that is skewed in some way about caring about people's welfare. What's the right one to use? Is there a way to answer that? Good question. So building on the analogy between individual decision making and societal well-being, as Harsanian Rawls do, I want to think about what principles should we use when making decisions for other people? and that will hopefully help us figure out what we should do in this case. So I think when making decisions for other people, we are as risk-averse as it's reasonable to be.

So we have these sort of ideas like don't take a risk for someone else unless you know they approve it or something like that. So if we can think about how risk-averse it's reasonable to be, in other words, how much. much weight could we reasonably put on, say, the bottom 10% of our distribution, whatever number that is, we're going to end up putting that much weight on the worst off 10% in society. Okay. So it does kind of fit together nicely, is the idea. Yeah. Yeah, exactly.

And since you published this, does everyone say, oh, yes, she's completely right. We agree? Or are there still critics out there who... Absolutely. Everyone thinks I'm right. That's good. Okay. if we could imagine hypothetically, someone might want to disagree.

Like, what would the, what would the criticism be? Yeah, so are you thinking of, there are different kinds of criticisms, both for the decision theory and for the social, the theory about societal distributions. For the decision theory. Let's, let's, let's, for the decision theory. Yeah. So a lot of the way this kind of debate goes is that there are these popular arguments, in favor of expected utility maximization.

And I try to show none of them work. They just end up with some weaker principle. Like I told you the argument about how things would go if we were to repeat the decision over and over again. I think that's not a good argument for expected utility maximization because we're not always going to repeat decisions over and over again. So all the arguments against my view are sort of of that vein. And one argument, which I think is pretty interesting, is about how people who are risk avoidant

in my sense will behave across time and how they'll respond to information. So I mentioned earlier in the podcast that if you're an expected utility maximizer, you're always going to want to make decisions on the basis of more information rather than less. but if you're a risk-weighted expected utility maximizer, which is what I call people who follow my view, then sometimes you're going to prefer to make a decision with the information you have rather than to get more information. And some people think that shows that risk-weighted expected utility maximizers

are irrational. Some people think, well, it's just a, it's just a, a theorem, a rationality that should all make decisions on more information rather than us. Can you give an example of where more information, I mean, is it more information is just not helpful or is it actively harmful? Well, the basic idea is that when you're getting more information, it's often not information directly that tells you. what's true in the case of your decision.

So for example, if you were thinking about taking a coin flip and you could get information about whether the coin would land's heads or tails, that's always going to be good. Sure. But there's going to be some cases in which information will talk you out of taking a risk that would have been a good risk to take. Okay. And those are the examples in which the risk-avoid person is not going to do better with more information. So in particular, you already have a high probability that the course of action you're thinking about will work.

And you're considering getting information that might put you into a state where you're like 50-50 about whether it will work. You're not sure it'll work. You're not sure it won't work. Those are going to be the kinds of cases. in which the risk-avoidant person isn't going to want more information. But your attitude is that there's no incompatibility between not wanting more information and still being rational. That's right. Yeah, that's right.

My attitude is sort of the more basic question is how should we take risk into account in decision-making? And then it's going to be an interesting upshot of the true theory, whether information is always good or whether. it's sometimes bad. Whereas my critics might say, no, it's part of the data that information should always be good. So however we take risk into account, we want our theory about that to respect this data. So maybe for the final question, has thinking about risk and rationality in this way changed your own way of thinking about risks? Or should it change anyone's way? Or is it merely a way of sort of reassuring yourselves that we're being rational?

Yeah, great question. Two things I want to say, and one is how we should think about ourselves, and one is how we should think about other people. So I got into decision theory, as I mentioned, because I'm really bad at making decisions. I hope there would be some mathematical theory that would just tell me what to do. Straighten everything out. After exploring the question of how to take risk into account, it turns out that even more is up to me than I thought. So the sort of upshot of that is that there's this additional thing that I have to think hard about, which is, do I want to be the kind of person that sort of like does pretty well no matter what? Or do I want to be the kind of person that goes for things where things might turn out amazing, but might turn out horribly?

That's something you might have to think about when you choose a job. It's maybe something you have to think about when you choose a romantic partner or a place to live. So the upshot is like, that's going to be a feature of you that you have to set. And there's like no right and wrong way to set it. You know, the sort of the romantic that's like, as long as there's a chance, I want to go for it, is doing something fine. And the sort of cautious person that's like, no, no, no, I just want to make sure things go better no matter what is also doing something. So that's the sort of personal upshot. The upshot for how we think about other people is that this is like an additional fact

that might vary between people. And just to put it in more concrete terms, when during COVID there were all these arguments about what restrictions we should have. For example, should we go out to restaurants or not? That was like a question people disagreed about. And one thing I noticed is that when people disagreed, they often thought the other person had really different values from them. So, oh, you want to go out to restaurants because you don't care about grandparents. You want grandparents to die.

That's why you want to go to restaurants. On the other hand, you had the person who didn't think we should have a lot of restrictions saying something like, well, you don't care about friendship. You don't care about people's livelihoods. You just want people to, like, be lonely and not have their jobs, right? So, like, they thought of each other as having different values. In decision theory terms, we would say, as having different utility functions. But if you notice that people can have different attitudes towards risk that don't show up in the utility function, you could sort of think, actually, these people had shared values.

everybody doesn't want grandparents to die. Everybody cares about that. Everybody cares about each other's work, each other's relationships and so forth. It's just that one group of people was weighing worse scenarios a lot more heavily than the other. So in fact, they were both being rational in their preferences, and they were both responding to the right values. It's just that they had these different risk attitudes. And now maybe it's a separate question of given that how that people have different risk attitudes, what should we do? But in making rules for all of us.

But the first thing is to realize that somebody might want to do things differently than you in one of these risky cases without having different values from you. Is this like one of the big five personality traits or something like that? Ooh, good question. Let's make it the big six and put risk attitude as number six. All right, that's good. We have a program going forward. That was very, very helpful. So Lara Bushak, thanks so much for being on the Mindscape podcast.

Thanks so much, Sean.