The a16z Show - Prediction Markets and Beyond

Starting point is 00:00:00 These markets are actually very good at producing predictions, which tend to be more accurate than polls. There is a sort of subtle distinction between wisdom of a random crowd and wisdom of an informed crowd. Instead of having politicians decide what policies to have, politicians and voters would just decide on what our metric for success is going to be. As you're deciding, like, which thing do we build first? or like as we're progressively decentralizing, like, what do we prioritize? You actually have to understand the market context you're working in. We as humans love making predictions. And to improve our predictive power, we've built up mechanisms that leverage the wisdom of the masses,

Starting point is 00:00:44 whether it be political polls, financial markets, even Twitter, or X community notes. One such mechanism that had its moment this year was prediction markets, with search queries for platforms like Polymarket or Kalshi, going hyperbolic ahead of the election. election. Now, today's episode is all about prediction markets, including where they're useful and where they're limited, but also how they coexist with other mechanisms like the polls. So was the attention they received an election year phenomena or a sign of something to come? And what's the difference between gambling and speculation anyway? And what implications does

Starting point is 00:01:20 that question have on their future in the United States? Given that this episode was originally published on our sister podcast, Web 3 with A16Z, we also explore where Web3 and decentralized networks play a role here. Finally, if you're excited about the next generation of the internet, be sure to check out Web3 with A16 and Z wherever you get your podcasts. All right, on to the episode. Welcome to Web3 with A6 and Z, a show from A6 and Z crypto about building the next generation of the internet.

Starting point is 00:01:54 I'm Sonal Choxi, and today's episode is all about prediction markets and beyond. Our special guest is Alex Tabarach, professor of economics at George Mason University, and Chair in Economics at the Mercatus Center and Scott Commoners, research partner at A6 and Z Crypto and professor at Harvard Business School. Prediction markets hit the main stage in the recent election, which we cover briefly,

Starting point is 00:02:16 especially to tease apart the hype from the reality there, since people have talked about the promise and premise of these for a very long time. But we also go more deeply into the how, why, and where these markets work, challenges and opportunities, including implications for designers throughout. We also briefly cover other information

Starting point is 00:02:34 aggregation mechanisms and discuss applications for all these markets, including touching on trends like putarchy, AI entering the market, D-Sai, and more. About halfway through, we discuss where do and don't blockchain and crypto technologies come in. And as a reminder, none of the following should be taken as business, investment, legal, or tax advice. Please see a6.com slash disclosures for more important information. Be sure to also check out the show notes for this episode. We have a really rich set of links, including all the research. research cited in this conversation. But first, we begin with a quick overview of what prediction markets are. The first voice you'll hear is Alex's, followed by Scots. So I think a prediction

Starting point is 00:03:16 market is a very simple idea. The bottom line is that we're interested in forecasting. Lots of people are interested in forecasting things. And prediction markets are some of the best methods of forecasting, which we have yet created. They tend to be better than complicated statistical models. They tend to be better than polls, or at least as good. And there's one reason for that. Suppose that a model was better. Okay, suppose I have the Nate Silver statistical model of predicting elections and it's better than the prediction market. I suppose that were true. Well, if that were true, I could make money. Yeah. I could. and use Nate's model to go and make bets.

Starting point is 00:04:02 And in making bets on the prediction market, I push the prediction market closer to the truth. So almost by definition, the prediction markets have to be at least as good. And typically they're better than other methods of forecasting. And actually, you know, sort of that is an illustration of why we think of these things as information aggregation mechanisms, right? What are they really doing?

Starting point is 00:04:24 They're aggregating information from all of the people in the market. And so if many different people, are out there doing their own private forecasts and like calibrating their own models. There's Nate Silver and like Jonathan Gold, Melanie Bronze, you know, we'll make up all of our variations. You know, they all have their own models. They all have their own estimates, which they trust with some degree of confidence. They come together, right? They're all buying or selling the prediction asset based on what their model leads them to believe. And so as a result, the asset is sort of like aggregating all of this information. It's price discovery, just like we think about in financial

Starting point is 00:04:56 markets and commodities markets. Like everybody's demand together discovers the price at which the market clears. And here, because what the value of the asset is depends on probability, right? It's like its value is sort of like a function of the probability of the outcome of the event. The price aggregates people's estimates of that probability. Yeah, exactly. I think it's useful that these are markets, and actually all markets do this. And we learned this going back to Hayek's 1945 article, the use of knowledge in society. This is a Nobel Prize winning paper, which doesn't have a single equation in it.

Starting point is 00:05:33 So anybody can go and read this paper, and they should. It's a fantastic paper. I'll link it in the show notes. Awesome. And so what Hayek said is, you know, prior to hire people thinking what prices, you know, the coordinate and they make demand equal supply

Starting point is 00:05:47 and production consumption. Hayek said, no, no, no. You're thinking about the price system all wrong. The price system is really about aggregating and transmitting information. And he said, look, there's all this information sort of out there in the world. And it's in heads, right? It's in people's heads, like what people prefer, their preferences, but also people know things.

Starting point is 00:06:07 They know what the substitutes are, what the compliments of a thing are, how to increase supply, what the demands and the supplies are. It's all in heads. And for a good economy, you want to use that information, which is buried in people's heads. But how do you get it out? Because it's dispersed. It's dispersed in millions of people's heads. the information, sometimes it's fleeting information, it's sometimes tacit, it's hard to communicate

Starting point is 00:06:32 to a central planner. So what Hayek said is that markets do this, because markets give people an incentive through their buying and selling to reveal this kind of information, to pull this dispersed information for millions of people and people who are buying, they're pushing the price up, people are selling, they're pushing the price down, suppliers, consumers, They're all in the same market. And so all of this dispersed information comes to be embedded in the prices. And kind of remarkably, the price can sort of know more than any person in the market. I just want to pause on that for a quick second because you guys might take it for granted,

Starting point is 00:07:13 but that's a very profound insight. Like what you're basically saying is that it's really surfacing what people know collectively at scale and getting at the truth in that way. I mean, that's a very profound thing. I just want to pause on that for a quick second. Exactly. What economists have found is that these markets are actually very good at producing predictions, which tend to be more accurate than polls.

Starting point is 00:07:38 So if you go to a prediction market, for example, the recent election with Trump and Harris, you can buy a asset which pays off a dollar if Trump wins and nothing if Trump doesn't win. Now, you think about how much are you willing to pay for that asset? Well, if you think that Trump has a 70% chance of winning and you go to the market and you see that the price of that asset is 55 cents, you're going to want to buy because you're buying something you think is worth 70 cents, 70% chance that Trump wins and you get a dollar. And you can buy it for 55 cents. So you expect to make 15 cents. And by doing that, you push the price closer to 70 cents so you can interpret the price as a prediction. and in the most recent election,

Starting point is 00:08:24 the prediction markets were tending to predict a Trump win even when the polls were closer to 50-50. Actually, the polymarket CEO said a lot of people trust the market, not the polls, at least when it came to the elections. Like, do you guys agree with that or no? I'm just curious, because if that's a place where we can quickly tease

Starting point is 00:08:40 some hype versus signal. I don't think polling is dead. Polling is one of the inputs into prediction markets. It's pretty useful. I do think people need to be more sophisticated about how they poll and who the poll. It's pretty clear that a lot of people now obviously are not answering their telephones,

Starting point is 00:08:57 and a lot of people don't want to talk to the pollsters. So there needs to be some new sophisticated techniques, but there has to be ways of drawing information from asking people questions. That's not going away. I saw on Twitter that landline poll response rates in the olden days were like above 60%, but today the response rate to like 5%, which means you're getting like a very bad sample bias in terms of who's willing to answer a call on a poll.

Starting point is 00:09:23 Like, I'll hang up right away if someone tries bowling me. Yeah, and in particular, it's not that like prediction markets will outmode polls. It's actually, they're going to lead to revolutions in technology for doing this well. If anything, like the availability of prediction markets increases the incentive to conduct polls, right? Like, you know, as we literally saw with the whale, you know, they went out and ran their own poll precisely because they thought they could use it usefully in this market. That's fantastic. I have to ask, though, so this may seem obvious to you. But the key point is that you're putting a price on it where people are putting skin in the game,

Starting point is 00:09:55 essentially, with their opinion or prediction, so to speak. And that seems very interesting and useful. How is that different from betting? I mean, can prediction markets be incredibly tiny amounts that don't have big value to be valid? Like, how does the pricing part of this all work in terms of the incentive design? Well, so at some fundamental level, the pricing works exactly as Alex described. If you think the probability that Trump is going to win at 70%, you see, the price of 55 cents, if you believe your prediction, you now have, you know, an incentive

Starting point is 00:10:25 to show up and buy. And like, you know, if enough people have beliefs of different types and they all come into the market and they all purchase, eventually the price sort of converges according to the convex combination of all of their different predictions. But when you ask about like the size of the market or the size of a betting market doesn't matter once people are there and they've already formed their opinions, but it might affect their incentive to gather information, for example. You know, if the size of the market is capped at $1,000 and you think the probability is 70%, you're not going to invest like $10,000 to get a more precise estimate, right? Like if the maximum possible upside for you is on the order of $1,000, you can't possibly

Starting point is 00:11:04 invest more than that to learn new information that will change your estimates and thus potentially sort of like inform the information in the market even more. And it's funny, I mean, we've been talking a lot in the wake of this most recent presidential election. about, you know, sort of prediction markets as having been very strong predictors in the trend direction of what actually happened. But, of course, if you look at, say, the 2016 election, that didn't happen at all, right?

Starting point is 00:11:28 You know, the prediction markets totally didn't call Trump, and they also didn't call Brexit, which happened sort of like, I think, the preceding summer or so. Oh, yeah, yeah. And, like, there people were asking, like, what happened? Like, how did these miss this? And at the time, I wrote an opinion column

Starting point is 00:11:42 where I argued that this information thing was a key part of the story, that, like, at least at the time, markets were relatively narrow both in terms of the total amount that could be, you know, sort of the total upside, the total amount that was enclosed in the market, and in terms of who participated in them, right? It was sort of like concentrated in a small number of locations. And those participants, because the upside was not necessarily that high, didn't necessarily have an incentive to go out and find out, you know, sort of like what's going on in other parts

Starting point is 00:12:10 of the country. And so you end up aggregating information just from the people who were already there, which might not be a good estimate in that circumstance. I want to push back a little bit on what Scott said. Go. Ooh, yeah, that's what I want to. Cards on the table, I am much more of a like prediction market bear than Alex is. We're both really excited about them, but there's a stack rank in our estimate of. Oh, funny.

Starting point is 00:12:31 Well, so I agree you want a thick market, of course, and it helps to have people willing to bet a lot of money because then they're willing to invest a lot in making their predictions accurate. The part which I want to push back on, however, is this idea that the market did not predict well if it predicted a 40% chance of Trump winning and Trump, you know, actually won, right? Because this is what people always do and frustrating, right? Because you can go back and look at individual examples and say, well, did the market predict well? But that's just like, you know, you flip a coin and it says 50% chance of coming up heads and it came up tails. and you say, oh, well, your probability theory isn't very good, is it? You predicted that they would have a 50% chance and it came up 100%.

Starting point is 00:13:18 So what's the real test? Well, the real test is you need a large sample of predictions, which could be predictions from political markets, but prediction markets predict other things as well. You need a large sample. And then you have to say, in the sample of cases in which the market predicted 40% a win, of, you know, the Republican or whatever. Of that, how many times did the Republican actually win?

Starting point is 00:13:44 And what you find is that pretty close, 40% of the time that the market predicted to win, 40% of the time Republic is actually won in those cases. So in other words, there's sort of a linear relationship that when the market predicts a high chance of winning, that happens a lot. When markets predict something with a low chance of winning, that doesn't happen very often. But, of course, sometimes it does happen, right? you know, something which happens with only a 5% probability ought to happen one every 20 times. And that's exactly what you see with these prediction markets.

Starting point is 00:14:17 They tend to be more accurate than other methods of forecasting, and they tend to be not systematically biased. We can talk about there's some odd biases, which are possible, but they tend not to be systematically biased. So it's not the case that something which is predicted 40% of the time actually only happens 20% of the time. the markets systematically get 40% of the time it's predicted, 40% of the time it happens. Let me give a simple non-market example, which I think illustrates this kind of a famous, people have heard of the wisdom of the crowds, right? And so you ask people, how much does this cow weigh? And people are not that good at, you know, figuring out how much a cow weighs,

Starting point is 00:14:56 some are too high, some are too low. But if you take the median prediction of how much the cow weighs, the median prediction tends to be very, very accurate. So in a sense, the crowd knows more than any individual predictor knows. And in the same way, markets do the same thing. The embed in the price more information than any single individual knows. Right. And just to be super precise, like you're specifically saying the median, not the mean, not the

Starting point is 00:15:25 mode. It has to be like the exact middle point, literally, not like averaging out from the extremes. In that particular example, yes. particular example, but it varies by context. Got it. Exactly. Let me build on that and like illustrated again, sort of like through a simple example, but in the language of the price system.

Starting point is 00:15:43 So when you're going around and polling people about the weight of a cow, you have to go around and ask them, and they don't necessarily have a strong incentive to figure it out. But suppose you have a very large amount of money to invest in commodities or commodities futures or something of the sort. And you have a predictive model that tells you, what you think is going to happen to these markets. Like, you have reason to believe that there's going to be a big shortage of oil

Starting point is 00:16:08 or a surplus of orange juice or something of the sort. You can buy and sell in the market in a way that reflects that estimate that you have, and it pushes the price accordingly, right? So if you think there's going to be a big shortage of oil, you're going to stockpile oil today. You're going to buy a lot of it today, and that's going to push up the price. Because, you know, suddenly there's more demand than there was before. And so when you see the price of oil going up, it's like it's a signal that somehow people think oil is more valuable right now than it was five minutes ago.

Starting point is 00:16:41 By the way, of course, these are all hypotheticals. Like, none of this is investment advice. Like, people should not go out and buy a bunch of oil or oil futures or whatever. But like, conceptually, that's how the price reflects the information. And the more strongly you believe that there's going to be a shortage, the more you're going to be willing to pay to buy right now. Right. And thus, the sharper the price movement, even sort of the stronger the inference about the information that that buyer brought to the market. Yeah. If you want to know whether there's going to be a war in the Middle East, keep an eye on the price of oil.

Starting point is 00:17:11 I remember that as a child in the 80s. And it's still true today. That's exactly right. And don't that the oil market is a little bit of a prediction market too. Right. The oil market is revealing information about people's beliefs about things that are correlated with the availability of oil. Yeah. Like whether there's a war in the Middle East. Well, that actually goes perfectly to the question I was about to ask, because I still want to dig a little bit more into the economic and market foundations, and then we can go more into the challenges of prediction markets and where they're going. But on that very note of oil, actually, great example, Scott, the question I wanted to ask you both is, where does this break? Because in the oil example, one could argue, well, it's not like a quote pure market. You have cartels. You have other horses that play. Now, you might be saying it doesn't matter because all that matters is people's opinions, which is what the prediction market is putting it. inputs into the market, or doesn't it matter? I guess my question is really getting at, what are the distortions that can happen here? Like, there are things that can manipulate it or other distortions where people's behavior changes so significantly that they untether the market from reality. Yeah, sure. I mean, one of the things about these markets, you know, oil predicting possibility of war in the Middle East, of course, they're not designed to do that, right?

Starting point is 00:18:21 In those cases, the information is sort of a leakage. It's an unintended consequence. of market behavior, which is very useful. You know, it's very useful for economists to be able to pull information out of these market prices. It's with the creation of prediction markets, which was really the first ones go back to the Iowa political prediction markets created in 1988. It was there almost for the first time that a market was created in order to produce information. Right. So there's a much more direct connection between.

Starting point is 00:18:57 the output of the market, the prices on the market, and the predictions, because that's what they were designed to do. Now, of course, you're totally correct that if you want to get a market to predict the future, you're going to want, as Scott said earlier, to have lots of people because you're going to take advantage of all the dispersed knowledge because, you know, there are people in Pennsylvania who have extra knowledge, you know, about what their neighbors are talking about, you know, that can give them a little bit of insight. right, that you might not have if you're living in New York or San Francisco.

Starting point is 00:19:31 So you want lots of people to participate and you want the markets to be quite thick because you want people to be able to want to kind of invest some time and energy, but the predictions should be that maybe apply some models perhaps to it, things like that. And of course, you want it to be free and open and you have to be a little bit worried about manipulation. Yeah. There are some like funny edge cases that we've seen crop up occasionally. In fact, there were even allegations that maybe that was going on here, where if there's some external outcome or even some like internal, like behavioral outcome that conditions on the prediction, right?

Starting point is 00:20:08 So if like political candidates are going to decide how hard to campaign in a given state based on what the prediction for that state says, you might want to influence the price not for the sake of earning money in the prediction market, although that might happen too, but rather because you just want to. place the prediction in a given position. Now, that's very hard to do because you actually have to change beliefs from doing that. In the, I think it was the Obama versus McCain campaign, somebody tried to sink a bunch of money to move the McCain percentage. And then, you know, people who had estimates that the Obama probability was higher just sort of arbitrage that out over a period of, you know, an hour or two, right? Like, you know, markets work. If you see something that looks to you like a market anomaly, you'd buy or sell accordingly. Yes, yes. I mean, we're all market periods here, so that seems like that's working. Yes. But if the market is thin or if the information signals are very dispersed,

Starting point is 00:21:00 maybe you can convince people, right? If you have enough money to like swing the market in a, in a very sharp way, especially if you're doing it through Sybil's, like many identities, if you're doing it through many identities, so it looks like a surge of people who have a given belief, you might actually change the beliefs of the market participants in a way that actually distorts the probability and could have various other impacts. And then the other thing is this idea that the oil markets are leaking information, we'll stick with that example, the oil markets leak information about potential conflict in the Middle East, right? That's a feature and a bug, right? The fact that it's an oil market that is informative about the Middle East, on the one hand, as Alex said, it means

Starting point is 00:21:40 that the market is not optimized for specifically answering the question what's going to happen in the Middle East. There's lots of other stuff that affects the oil market, like, you know, how popular electronic vehicles are at that given moment in time, right? So you have this very complicated signal extraction problem, right? You see a big spike in the oil price. Is it because there's a potential conflict coming in the Middle East? Or is it because there's just been like some new electronic vehicle test that failed? And like somebody knows that. And so they know that oil is going to be more important next month. Whereas if you have a market that's just predicting, will there be a conflict in the Middle East? That's all it's predicting. But of course,

Starting point is 00:22:14 that's now a zero-sum market. It's sort of a harder market to participate in if you only have dispersed information, right? If you don't actually know where, you don't actually know whether there's a conflict in the Middle East forthcoming, but know that some things that are happening like sort of suggest that. For example, you saw an oil price change. You have to do a much more complicated and you're taking a slightly, in some ways,

Starting point is 00:22:34 a riskier bet by participating in a prediction market where you're staking everything on this one outcome rather than on something that's like heavily correlated, right? Where there are many different things that could have related, you know, sort of related predictions could be mostly correct even if your main prediction is wrong. So the takeaway is prediction market's narrowness

Starting point is 00:22:52 is a feature in a bug, it's sort of dual to the sense in which ordinary market sort of broadness is a feature in a bug, right? Because a prediction market is a narrow

Starting point is 00:23:02 zero-sum contract on a specific event, many people's information about that event is actually coming from all these correlates. It's not that they know specifically,

Starting point is 00:23:11 like, is there a conflict coming in the Middle East? They see a lot of potential signals of it. And so if you're buying and selling in a market that responds to those signals, that sort of like, ensures you a little bit, right? If you get the main estimate wrong, but all your signals were correct,

Starting point is 00:23:27 you know, you're at less risk than if you go into a prediction market and had all the signals right, but the final estimate wrong, and then, you know, you're just betting on the wrong side of the event. I think what Scott said also has implications for, why don't we have prediction markets and everything? I mean, if these markets are so great and they work so well at predicting things, you know, why don't we have more of them? And I think Scott was basically giving the answer there. This is how I would put it. You know, if you have the market for oil, then there are lots people who are buying and selling oil who are not interested in what's going on in the Middle East. Okay. They're not trying to, you know, predict that, right? But it's precisely because

Starting point is 00:24:03 you have lots of sort of organic demand and supply that this provides a subsidy to the Sharks who go in there in order to make the price more accurate. Or take the example of, you know, wheat. There are lots of farmers who are buying and selling in the market for wheat just to insure themselves, just to hedge themselves. And it's because of that native organic demand that the market is thick enough that you then have all of the sharks who are not themselves farmers, but they go in there and they use models and techniques and whatever to predict which way the market for wheat is going to go and they make that market more accurate. Now, if you didn't have the organic demand, then you're going to have a market with just sharks in it. No farmers and

Starting point is 00:24:56 just sharks. And who wants to be in a market where you're only with other sharks? If I know that the other guy is just trying to predict this one thing as much as I am trying to predict it, you know, I don't want to be in a market with Scott. He's just too smart. Right. I would say the same thing about you. And that's why the market wouldn't work. That's why the market wouldn't work. Right. So some of these markets, even though they might be forecasting something which is useful, there isn't enough organic demand where you have to subsidize it from outside the market in order to get a useful prediction out of it in order to get the shorts willing to go against one another to try and predict this thing. And that's why we don't have markets and everything yet potentially. This is maybe jumping ahead a little bit, but I just have to ask at this point, I mean, Scott, you're like a market design expert. So on the market design front, what does that mean if there isn't?

Starting point is 00:25:48 organic demand, is there a way for market designers to essentially create markets in situations where there isn't that kind of latent organic or existing thing to harness? Like, can you actually manufacture that market without distorting it and kind of create conditions that could design a market into place? That's a great question. I mean, there are two different ways to get at it. One of them is, which is sort of what the framing of the question is pointing at is, could you find a way to create latent demand?

Starting point is 00:26:16 And Alex was saying you could subsidize it, right? you could basically like somehow subsidize the experience of some people trying to predict this event. Like, you know, subsidize a bunch of college students developing forecasting models. So then they have a lower cost of entering the prediction market or something of the sort. Again, not advocating this specific policy. Right. Although in Alex's example, that subsidy was not intentionally a subsidy. It's just a result of the behavior.

Starting point is 00:26:38 Like it wasn't like people are trying to subsidize. It was just subsidizing because of their natural behaviors. True. No, no, exactly. So like we started this conversation with the recent presidential election. and all of these other associated elections, those have proven, at least in practice, to be much thicker markets,

Starting point is 00:26:53 because there are some people who seem just interested in betting on them, right? A lot of people have some amount of information, some amount of opinion. And so there's a little bit of that latent demand that sort of comes from people's general interest in the question. Yeah.

Starting point is 00:27:05 One could try and create that for other contexts, right? You can try and help people feel that something is interesting or feel that they have an opinion about it enough that they're willing to participate in a prediction market. The other thing you can do is you can use other types of information elicitation mechanisms. Prediction markets are one of many ways of doing incentivized information aggregation.

Starting point is 00:27:25 And others are things like incentivized surveys or peer prediction mechanisms. There's a whole class of what are called peer prediction mechanisms where what you're in effect doing is asking people what they believe about an outcome and what they think other people will believe about an outcome. And then you use sort of their beliefs about others as a way of cross-examining whether they were telling the truth. because you survey a lot of people, you get sort of like your crowd volume.

Starting point is 00:27:50 You sort of know the aggregate belief of the population, and you can check whether someone's own belief about the population is sort of the right mixture of that aggregate belief and their belief. So like, if you yourself think that Trump is more likely to win,

Starting point is 00:28:06 then you yourself are more likely to believe that other people think Trump is likely to win because the frame you have, your information sort of like indicates that at least one more person, at least one person in the market believes that. And so one can cross-examine your predictions with your estimate of what the population believes and what the population actually reveals that they believe.

Starting point is 00:28:27 And then you can reward people based on how well they did. In effect, how good are you at estimating what everyone else thinks given what you think? And those sorts of mechanisms you can incentivize you can pay people immediately, incidentally. I think prediction markets where the event has to be realized, the payments only realized at the end. Here, you're not paying people based on their accuracy about. the event, you're paying them based on your accuracy about everyone's estimates. And so you can do that all at once, right? Collect all the estimates, pay people, they go home, you have your estimate. And these have been shown in practice to be very effective for small population or like opinion

Starting point is 00:29:00 estimates, things where there isn't a thick market and like a very, very big public source of signal. That really answers that question. And by the way, it brings up a very important point that we did not address in the recent example of the election, which is the French, quote, whale, who won by using the neighbor poll where, you know, their neighbors won't say what they think, but when you ask them, like, who do you think your neighbors are going to vote for? It's kind of a way to indirectly reveal their own preferences, and that's the so-called neighbor poll.

Starting point is 00:29:28 I don't know if that's a standard thing or that just came up in this election. It's the first time I heard of it. But it's a great example of something I did study in grad school when I was doing ethnography work, which is never trust what people say they're going to do, but what they actually do. This goes to your economist's world of revealed preferences. Absolutely.

Starting point is 00:29:44 Right. Very similar. But anyway, in that case, that person pulled his neighbors and then used that data, essentially off-chain, to then go back onto the market, Wally Market in that case, to up his bet, and essentially won big as a result. So, like, that would be an example of what you were mentioning. Although in that context, you were mentioning it in how can we address a case where there's a thin market, this is a case where that played out in a thick market of the election. Well, you might say that he was using this in the thin market of trying to understand his neighbors, sort of like local preferences and estimates.

Starting point is 00:30:16 There you go. That's more precise. Although we actually don't know the details of how he produced these estimates. It doesn't sound like they were incentivized. So it's not exactly like what I was talking about with peer prediction. But you're right. It's the same core idea that like using people's beliefs about the distribution can be much more effective than using their personal beliefs a lot of the time. So I would underline two things there.

Starting point is 00:30:37 One, yeah, the market is a way of bringing all of this dispersed information and creating an aggregation. but it's not the only way. That's kind of what Scott is saying, right? And understanding, this is one of the first information aggregation mechanisms which we have studied and understood reasonably well, but there are other ones. And so you can think about prediction markets as being one example of a class of mechanisms which take dispersed information and out of that pull some knowledge, which none of the people in the market are that none of the people you polled, none of them might be aware of it, and yet somehow it is in the air, as it were.

Starting point is 00:31:18 That's fantastic. There are also other ways of subsidizing these markets, which is something that corporations may be very interested in doing because corporations are interested in forecasts in the future, and some of them in the past have created their own internal prediction markets. So one famous example, of this is Hewlett Packard. They were interested in forecasting how many printers are going to be sold in the next quarter and the next two quarters, three quarters, four quarters, and so forth. So they created a market where if you correctly predicted how many printers would be sold,

Starting point is 00:32:02 in which time period, you could earn money. And they subsidize that market. So everybody going in, which is just HP employees, got like $100 to play with. So that's a way. And, you know, of trying to get more people involved and interested in playing on these markets to elicit disinformation. That example is actually really interesting to me because when I was at Xerox Park, we talked about that. And one of the things that came up is it's a very useful mechanism to your point, Alex, for getting certain things right. But it is not a useful mechanism for actually figuring out the future in terms of what to invent because it doesn't address the case of you don't know what you don't know. You only know what you know. And this came up just yesterday,

Starting point is 00:32:42 Trump announced his candidate for Attorney General. And one of the examples someone cited on Twitter was it's the first time they've seen a polymarket contract resolved to zero for all potential outcomes because Gets wasn't even listed among the 12 potential nominees in those range of possible outcomes. So that's an example in that case where you have to have the right information itself in that prediction market. And maybe you guys can explain that a little bit more too really quickly because I think that HP example is super interesting on multiple levels.

Starting point is 00:33:12 Yeah. So these markets are good at when you figure people have got some knowledge and it's hard to aggregate that knowledge. The other thing they're good at, you know, the people have run these markets for predicting when a project will be complete, right? And this is a classic case. When you ask people, they're going to be, oh, no problem, it'll be ready. But, you know, in five weeks, well, you know, whatever, right? They're very optimistic. And yet they tell the boss it's going to be ready in five weeks. Well, they go back and tell their friends, oh, my God. it's delayed, and we get all these problems. But if you let people bid anonymously in these markets, then the truth comes out. So this is a way of the corporate leaders can learn information that their employees know, but are not willing to tell them, right? But to your larger point, yeah, I mean, nothing is more difficult to predict in the future. Right, right. And, you know, Trump is a chaos. agent, right? Whatever he's going to do, like, it is hard to predict. And I agree. I don't think anybody predicted Matt gets. Well, and indeed, actually, so this sort of highlights, you know, we were

Starting point is 00:34:22 talking about what prediction markets are good at versus where you might want to use other sorts of information elicitation mechanisms. The two examples that Alex gave of within company prediction markets, you know, predicting sales or sales growth or something that's like, you know, a metric that many people in the firm are tracking and have different windows of information into, predicting when a product is going to launch where, like, you know, you might have product managers who know something, you might have engineers who know there's a hidden bug that they haven't even told the product managers about yet. Again, it's like, these are contexts where many of the people in the company have some information

Starting point is 00:34:55 that only they have and that the aggregate of all that information is a pretty good prediction of the truth because the actual outcome is the aggregate of all those people's information directly, right? It's like, how many sales calls are you making that are succeeding? Or, you know, how is the coding for this specific feature going? By contrast, you mentioned with Xerox Park, you know, trying to predict whether a new sort of totally imagined product is going to succeed, well, that's really, really hard. And it doesn't rely on information in particular that the company has, right? Like, yes, the company has some idea of what products

Starting point is 00:35:26 people might buy, but you might be like, you know, AT&T and invent the first picture phone or something of the sort. And, like, you thought that was a great idea, but you don't actually know until you put in the market and see whether people are, like, interested in using it. And so the aggregate of all the information in the company, there there's a product they went through with, right? They concluded was a good idea based on all the signal that everyone in the company could see and it still flopped. The total information in the company wasn't high enough

Starting point is 00:35:51 to actually, like, provide the right answer, even when aggregated. Right. But I do think there is a sort of subtle distinction between wisdom of a random crowd and wisdom of an informed crowd, right? Like, again, with our Hewlett-Packard example, Hulet Packard sort of knows that if you're trying to figure out now, like, you know, whether a product to launch on time, a random person on the street has no information about this. You don't want to like pull together a focus group of miscellaneous Hulet Packard customers and ask them, when do you think we're going to finish designing our new printers, right? I don't know, like you've released a printer last year, probably next year, maybe? Who knows?

Starting point is 00:36:25 And so there is this question, are you learning things from the right crowd? You know, you could have the best incentivized information elicitation mechanism on the planet. And if you only survey people who don't know anything at all about the topic, you're incentivize them. You'll learn what they believe truthfully, but you won't be able to do anything with it. Yeah. And then back to the future, like the whole idea, the best way to, you know, predict the future is to invent it. Like that goes just like the jobs and the, you know, the phone. Like no one, you can ask a million people, will they ever use a touch phone? People's behaviors can also evolve and change in ways that they themselves are not aware of, which is that other, that example. Yeah, prediction mark, it's like a candle in a dark room, right?

Starting point is 00:37:05 I mean, it helps us see a little bit, but there's still areas which you can't see very far. Great. I'm going to ask a couple of quick follow-up questions from you guys so far. So just to be super clear. So thin versus thick. You guys are talking about the depth of the market, like in terms of the number of participants. Thin is too few. Thick is many?

Starting point is 00:37:22 Is that correct? Or is there a better, more precise way of defining that? Yeah. So, I mean, in the prediction market, a thin market is few people betting small amounts. And in fact, one of the problems we've had is that prediction market, are mostly illegal in the United States. So the biggest one in this past election was polymarket, which it was illegal for U.S. citizens to bet on that market.

Starting point is 00:37:47 We're slowly changing, but we do have this kind of ridiculous situation. I think it's ridiculous anyway, that we have huge markets in sports betting, gambling, right? Huge, huge markets. And we allow that. And yet here we have a kind of gambling market, a prediction market, where the output is actually really quite useful. It's quite socially valuable, and we don't allow it. So making these markets legal and open to more U.S. citizens would thicken those markets, make them more accurate, attract more dispersed information,

Starting point is 00:38:22 and I think would be really quite useful. But to your bigger point, Alex, you're basically arguing that they can be a public good in the right context, informationally. Absolutely. And interestingly, if you think about. some of these prediction markets that are getting served notices and whatnot. And we don't know why, to be clear, but it's interesting because in some cases, people might argue some people trying to get information is a manipulation of the market. But in fact, to your guys' entire point throughout this discussion, it's actually ways to provide more input of information into the market

Starting point is 00:38:53 itself, too. So that's kind of an interesting point on the public interest side. Let me give you another example on this public good nature of these prediction markets. One of the most interesting, fascinating uses of these prediction markets is to predict which scientific papers will replicate. Oh, yeah. You know, we have this big replication crisis in the scientists, psychology, and other fields as well, of, you know, lots of research and it doesn't replicate. Well, what some people have done is it's expensive to replicate a paper, but one thing people have done is to have a betting market, a prediction market, in which papers will replicate. And that's a very important. And that's That turns out to be very accurate.

Starting point is 00:39:33 And then you only have to replicate a few of those papers in order to have the markets pay off. And for the rest of them, you use the prediction market result as a pretty good estimate of whether it will replicate or not. So this is a way of improving science, making science better and quicker and more accurate. I love that. I brand a lot of op-eds when I was at Wired on Open Access and Science and kind of like evolving, you know, peer review and replication crisis and the whole category and theme. So it's very exciting to me to hear that that's something that we can do to address that. It leads to a quick follow-up question, which actually happens to be on my list of follow-up questions for you in the lightning round of this, which is when you guys were talking earlier about this just kind of tapping into this intuition information dispersed across many people into these prediction markets, one of the first questions that came to mind is do you need domain experts or does that actually distort a market? And this actually comes up as a perfect segue from your point, Alex, that example of scientific papers.

Starting point is 00:40:30 because that's a case where one would imagine that people in that industry or that domain or just other scientists who have the experience of analyzing research would be the best at predicting things. But is that necessarily true? And do we have any research or data into domain expertise in these markets? I don't know the answer to that last part. Let me talk about the first part because it also speaks to your thick versus thin. Great. Yeah, good, good. So when Alex said a thin market is small number of participants betting small dollar amounts, why is that a thin market?

Starting point is 00:40:58 it's because the total information is small in two ways. One is that there are a few people bringing their own individual estimates. You just have like a small number of people saying things. And second, because they're betting small dollar amounts, it's sort of a signal that their information is not very like strong signal or confident, at least relative to what it could be otherwise. You know, if you are staking a very large amount of money on this, the market inference is that you have done the research, you know,

Starting point is 00:41:26 and indeed you have the incentive to do their research. You know, why is the inference that you've done the research? It's because if you're staking a large amount of money, you should have done the research. Because otherwise, you know, you're putting money at risk without sort of full information. Like the French whale, who did the neighbor poll to find out. Right, exactly. And one can argue about how good or bad that new poll was or whatever, like whether he should have trusted his information that much. But it's unambiguous that part of his confidence, and he said this, part of the confidence that he had to make that huge bet was that he thought he had a signal that was accurate and the market had missed.

Starting point is 00:41:56 And so, like, thickness and thinness, like the proxy for it, the way we think about measuring it is how many people and how much are they staking, like, how much value are they putting behind their beliefs? Thickness and thinness is really in terms of the information. It's like, do we have a lot of different signals of information that are strong coming together and mixing to determine the price? Or is it really just like a very small number of pretty uninformed signals? That's this tension when Alex is saying it's a problem. it's a problem that the biggest prediction market if the U.S. election was not actually in the U.S. and was not legal to participate in the U.S.,

Starting point is 00:42:32 well, yeah, a lot of the information, a lot of the, like, real signal is in the United States. And so without those people being able to participate in the market, you miss at least sort of a lot of that to a first order, right? You know, people internationally will be figuring out ways to aggregate and sort and try and use it, but, like, you miss a lot of the people who have that information already at their fingertips. And so you ask about domain expertise.

Starting point is 00:42:54 it's not exactly domain expertise versus not, but rather information richness. And for example, in predicting scientific replication success or failure, domain experts are especially well equipped to do that, right? Like a random person chosen off the street, you know, you can tell them a scientific study and maybe they'll have an instinct one way or another, whether they think they believe it.

Starting point is 00:43:16 But like a lot of the detail of figuring out whether something will replicate comes from knowing how to read the statistical analyses, trying to understand the setup of the experience and like the surrounding literature. And so their domain experts have a particularly large amount of information. If you think about something like a political betting market, maybe domain experts who are focused in the world of politics and polls and so forth have like a big slice of information.

Starting point is 00:43:39 They do. But there also might be other categories of people, like people who know that their neighborhood has like recently switched its political affiliation in a way that isn't yet captured in the national polls. Or are French Whale who went and ran his own sort of poll using. a custom chosen method. And so the context of the question the prediction market is trying to evaluate, and this is actually true for any informational incitation problem, this is just about prediction markets, right? The context of the type of information you're trying to learn tells you something

Starting point is 00:44:09 about who has the most information to bring to the market and thus who it's important to have there. Yeah, I agree with everything Scott said. One of the interesting things is you often don't know who the domain expert is, right, until after the market. is being run. So, of course, it's absolutely true that, you know, if you're going to be predicting political events, you want people who are interested in politics. If you're predicting scientific articles, people need to be able to read stats and things like that. But one of the guys in the scientific replication paper on markets, he made like $10,000 was just one of these super obsessive guys, right, who just really got into it and, you know, was running all kinds of regressions and

Starting point is 00:44:51 doing all kinds of things and stuff like that. And so when you say domain expert, I think one of the virtues of these prediction markets is that they're open to everyone and they don't try and say, oh, no, only the experts get to have a voice, right? It's more only ex post do we learn, hey, who really makes a money in these markets?

Starting point is 00:45:13 Right, right, right. Absolutely. I'm so glad I asked you guys about the definition of thick versus thin because you guys gave me so much interesting nuance to that. Because people I think following this podcast definitely understood what you meant about thin versus thick early on, but you guys just took it to a new level. If you're so smart, why aren't you rich? Hey, I am rich. I made some money in this market.

Starting point is 00:45:34 Well, and that again, that's about the incentives. We talk about like the dollar value staked, like the amount of money someone is staking on their prediction. Again, in equilibrium, it should be a measure of their confidence, how confident they are in their own beliefs and how much effort they've put in to learn the information to be precise. And so, Exactly as Alex says, one person who might be really good at predicting a scientific replication failure is someone who works in that exact same area. Another one, it might be someone who just like enjoys doing this for fun and like has never had a real incentive to triple down on doing it, but now suddenly they can't. Right, right. And by the way, Scott, does it have to be dollar and price incentives? I'm asking you this question specifically because you and I have done a lot of pieces in the past on reputation systems.

Starting point is 00:46:17 And I almost wonder if the skin in the game can just be karma points and not even any. money because I think from a pride perspective. 100%. So like Alex mentioned subsidy, right? Like one way that you can subsidize, I think he said Hewlett-Packard subsidized by giving all their employees $100 and saying spend it all on this market. You can subsidize people with cash, but you can also subsidize them with tokens or, you know, reputation or like, sort of various other sources of value. And one of the advantages of using tokens is that that way you can deliver a subsidy that's sort of only useful in this market, right?

Starting point is 00:46:52 You know, if it's like a personal non-transferable token, but I give you a bucket of them, and the only thing you can do with it is use it to enter predictions and you just choose which prediction markets you choose to enter into and how much you spend in each one, right? And then you earn payoffs.

Starting point is 00:47:06 Payoffs are also measured in tokens and maybe downstream, you might get prizes for having large numbers of tokens or something you get to join the elite predictor force or even just serves as a measurement of your reputation, how good you are at making predictions, which maybe you leverage into something else, right? Like people who win data science contests

Starting point is 00:47:20 leverage that into data science jobs. Maybe you leverage this into a forecasting job or something. All of that, so long as you find people who are willing to be incentivized by those types of outcomes, you can subsidize their participation in a unit that locks them into the market, right? That their one thing to do with it is to participate in the market and reinforces more and more participation

Starting point is 00:47:43 among the people who are most successful and most engaged. That's super interesting. And I'm going to push back on you on that, actually, because I actually wonder if it necessarily needs to be crypto-based and you can just do any kind of- No, no, it's any like internal marker. But for all the reasons we normally know, like, it's much better to do this in an open protocol form. Because for example, if the token is eventually going to be leveraged for reputation, you want anyone to be able to verify that you have it. Right. Audit it. Hence, blockchain's got it. Great. And we'll talk a little bit more about that, just more lightning questions. So where do super forecasters like Philip Tutlock's work come into all of this? Like are they especially good at prediction markets? Because that's a case where they're like generally better at the general public and sort of quote forecasting and making predictions. Is there a place for them in this world? Or are they kind of the outliers here? Or does it not even matter here? I think there's two things. One, I think

Starting point is 00:48:30 the basic lesson of Tetlott's work is most people, even the ones who are in the forecasting business, are terrible forecasters. Right. I mean, he first started tracking so-called political experts. And seeing what their forecasts were, you know, 10 years later, were they right, every five years later? And they were completely wrong. So he then shifted into looking for, is anybody ever right, are they super forecasters? And yes, he found that some people, you know, not typically the ones in the public eye, but some people can definitely forecast better than others. One of the things those people can do is then participate in these markets. And by their participation, they push the market price closer to their predicted probabilities.

Starting point is 00:49:19 So forecasters have an incentive to be in these markets. And by being in these markets, they make the markets more accurate. Now, is the market always going to be more accurate than the super forecaster? No. I mean, Warren Buffett, you know, he has made a lot of money, even though markets are basically efficient. But Warren Buffett has shown that he, in many cases, is able to predict better than the market price itself and more power.

Starting point is 00:49:44 to him. And so there are going to be some super forecasters, but they're hard to find. They're rare. And a virtue of the price is that everyone can see it, right? It's public. So this actually gets at a bigger, maybe more obvious point to you guys, but a recurring theme I'm hearing is it's not that the prediction market is only taking in like guesses and people's intuitions and bets and opinions and any information it has. But theoretically, done well, it's taking in all information. It could be super forecasters contributing to it. It could be Nate Silver taking his 80,000 simulations and feeding his inputs and adding that signal into it. It could be people who are pollsters, putting their data and predictions.

Starting point is 00:50:28 Basically, it doesn't even matter how people get at their intuition. All that matters is that they're pricing that information into that market, essentially. Do you know the Wall Street Betts is a famous, everything is priced in the post? No, I don't actually. I don't know this one either. Let me read it just a little bit. It's a fantastic post. like five years ago. It's called Everyone is priced in and he says, the answer is yes,

Starting point is 00:50:49 it's priced in. Think Amazon will beat the next earning? That's already been priced in. You work at the drive-through for Mickey D's and found out that the burgers are made of human meat? That's priced in. You think insiders don't already know that? The market is an all-powerful, all-encompassing being that knows the very inner workings of your subconscious. Your very existence was priced in decades ago when the market was valuing standard oil's expected future earnings based on population growth. That is so great. Okay, you have to send me that link, Alex, and then I'll put it on the show notes.

Starting point is 00:51:24 So you're basically agreeing that it's the markets do price everything in. Yeah, I mean, that's an exaggeration. But yeah, I mean, anything is fair game. I want to push back, but we're fine here because anything is fair game, but you have to wonder who's going to show up to those markets and where their signals are coming from, right? Like, if you're a super forecaster, maybe you work for like a super secretive hedge fund. And the last thing you want to do is directly leak what it is you believe. Yeah, yeah.

Starting point is 00:51:53 And in fact, you would prefer that the market be confused by this public signal. We talked about manipulation. You might show up and tank the prediction in one direction or the other just to take advantage of that in the financial market off to the side. And so while in principle these things can be very comprehensive, you still have to think about who participates in which market where. just like we see in other markets where like some people trade in darkpool, some people trade in public exchanges, and that selection sort of affects

Starting point is 00:52:19 what information price is really aggregating where. That's fantastic, yeah. The other thing about public forecasters, super or otherwise, is that they're very salient to the average person. And so another thing we see in prediction markets is herd behavior, again,

Starting point is 00:52:37 just like we see in other types of markets. Like, you know, if a lot of people are suddenly, buying oil futures, does that mean that they all have knowledge that there's going to be a conflict in the Middle East? Or does it mean they sell other people buying oil futures and are like, oh gosh, like, I'd better do this too? Or, you know, did they see one analyst report and they all saw the same analyst report? And as a result, they all went and bought oil futures because they believed the report. Or worse, did they see one analyst report that said, you know, like oil is going to be

Starting point is 00:53:08 expensive next quarter? And they went and bought oil futures. not because they believe the report. Maybe they even have information that it's not true, but they know everyone else is going to see the report. And so there will be purchasing pressure. Yes. There's this very famous paper by Morrison Shin in the American Economic Review

Starting point is 00:53:25 called Social Value of Public Information. Okay, I'm going to put that in their show notes. It talks about information hurting, right? The idea is basically, if you have a market where everyone has private signals and then there are some very salient public signals, and people have to coordinate, right? Are you going to run on a bank or not?

Starting point is 00:53:41 Or like, what do you think is the probability of this thing happening, people might ignore their private signals if the public signal is strong enough that they think other people are going to follow it. Yes. And so when a very prominent forecaster makes a prediction, like as the sort of polls were coming in in the week leading up to the election, a new major poll would drop. And then the prediction markets would jutter around and sort of veer off at least briefly in the direction of that poll. And that's this like public information effect, right? This is salient. You expect a lot of market movement based on this information. And so the market actually moves even more. It incorporates not just the information, but also the

Starting point is 00:54:18 fact that other people are incorporating the information too. And are there any market design implications for how to avoid that happening? Like if you're setting up the conditions of a perfect great prediction market? Oh, gosh, that's a great question. I mean, first of all, it's not completely avoidable. You can't have a market where a sufficiently strong public signal doesn't generate some herd behavior, right? It's just at that level it's unavoidable. But you can try and do things to dampen the effect. Off the top of my head, I can think of two. There are probably others. One is you could basically like slow trading a little bit, right? You could sort of like limit people's abilities to enter or exit positions very, very quickly. So it sort of forces people to like average.

Starting point is 00:54:59 Well, it's also kind of an example of slowing contagion, right? Like an infection spreading very fast. Totally. Kind of like the herding becoming viral. Yeah, contagion is a very good example of what's talking about, you know, in stock markets, we have circuit breakers. Circuit breakers. Yes, yes, yes, exactly. Circuit breakers. There we go. So that's one of the ways.

Starting point is 00:55:17 Another thing you could do is try and refine your market contracts in a way that orthogonalizes, by which I mean, it sort of extracts out the signal that is independent of that signal, right? So a prediction market contract somehow incorporates the information, sort of like, adjusted for whatever Nate Silver claims. Let me give you an example, because my colleague, Ron, Robin Hansen, who is one of the founders of prediction markets. Right.

Starting point is 00:55:43 Robin is usually many steps ahead. He is a very clever proposal for this, which I don't think anyone has ever implemented. But he says, you have a prediction market. And then you have a second prediction market on whether that prediction market will revert in the future to something else. Yes. Oh, so genius. Yes, exactly. That's the way you do it.

Starting point is 00:56:00 That's the way you're orthogonalized. Perfect. That's way better than my example. That's so great because I was actually going to guess something like combining the reputation thing. And this is essentially. a way of combining reputation by having a parallel market that verifies and validates. Exactly. Totally. That's so interesting. By the way, that's not Futarky, right? His new thing? One of the criticisms of Futarky was precisely the point which Scott made. And then Robin's response to that is,

Starting point is 00:56:25 well, the solution to a problem of Futarky is more futarchy. Ah, okay. And by the way, you quickly define Futarky for me? Yeah. So Robin Hansen's idea is let's take these decision markets and apply them to government. Let's create a new form of government. You know, there aren't many new forms of government in the world. Democracy, monarchy, you know, futarchy is a new form of government. And the way it would work is that instead of having politicians decide what policies to have, politicians and voters would just decide on what our metric for success is going to be. So it might be something like GDP would be one metric of success, but you might want to adjust it for inequality or for environmental issues. So you're going to create some net statistic GDP

Starting point is 00:57:15 plus. Then, anytime you have a question, should we pass this health care policy, how should we change immigration rules? Should we have this new immigration rule? You have a market on whether GDP plus would go up or down if we pass this new law. And then you just choose which one, if GDP plus goes up, you say, okay, we're going to do that. And so people would just submit new ideas to the futarky. Here's a proposal for immigration. Here's a proposal for health care. Here's one for science policy. And then you just run a prediction market. Would GDP plus go up with that or would it go down? And then you choose whichever comes out. So Robin expands this idea of decision markets to an entirely new form of government.

Starting point is 00:58:04 That's fascinating. And it relates so much to one of our partners, collaborators, work, Andrew Hall at Stanford. He studies a lot on chain and kind of liquid democracies and more. That's very interesting. Thank you for explaining that, Alex, because I've actually never fully gotten what Futarki is. People toss it around. And I'm like, but actually, what is it? I still don't get it.

Starting point is 00:58:22 So that was very helpful. It also sounds like it could be the subject of like a Borges short story or something. Oh, my God. Yes. Yes, absolutely. Oh, gosh. What was the last one that we put in the last reading list, Scott, for the Founder Summit? Was it the labyrinth?

Starting point is 00:58:35 I think it was labyrinths, right? Yeah, yeah, I think so. That's so funny. So a few more questions, and I want to switch to crypto. So since we're talking actually about like kind of market theories and practice in this recent segment, Alex, did you want to say a little bit more about efficient markets? Sure, sure. So another fascinating example of how markets could leak information, which then could be used for other things,

Starting point is 00:58:58 is if you ever seen the movie Trading Places, you probably know that the main determinant of orange juice futures is what the weather is going to be in Florida. Of course. So Richard Roll, who is a finance economist, had this interesting question. Well, can we use orange juice futures to predict the weather? And what he found is that there was information in those market prices, which could be used to improve weather forecasts in Florida. Kind of an amazing example.

Starting point is 00:59:28 Because no one, again, knew this. No one was even predicting this, but this was kind of a, leakage of this amazing information. Fantastic. Another fascinating one is, you know, Richard Feynman famously demonstrated that it was the O-rings, which were responsible for the challenge disaster by dipping the O-ring in the ice water at the Congressional Committee.

Starting point is 00:59:52 However, economists went back, and when they looked at the prices of the firms, which were supplying inputs into NASA and to the Challenger, they found that the stock price of more than thiochol, which was the firm which was produced the O-rings, that dropped much more quickly and a much larger amount than any of the other firms. So the stock market had already predicted and factored in that it was probably the O-rings, which were the cause of the Challenger disaster even before Richard Feynman had figured this out. And by the way, it's another that ties back to your HP example in a way, because if I recall, part of the backstory with the Challenger was also that it was a case of

Starting point is 01:00:35 death by PowerPoint because of the way they were communicating information internally and that the format and the structure kind of constrained how that information was presented. I think Tufti gives a famous case study of this in one of his many books. So another way of putting that, actually, which is kind of disturbing. But I think you're right in that the people on the ground, they knew this wasn't a good idea. They knew it was not a good idea. to launch the Challenger on such a cold day. And if there had been a prediction market of like what's going to happen or should we do this,

Starting point is 01:01:10 then I think it is quite likely that that dispersed information, which no one was willing to tell their bosses, you know, no one was willing to stand up and say, we should not do this. Instead, it got buried in PowerPoints. That dispersed information might have found its way to the top if there had been a prediction market in,

Starting point is 01:01:28 is this launch going to go well? Exactly, or said another way. The earlier definition of a prediction market, it would have been another way for management to elicit better information from their employees. And using just that as a mechanism for communication, essentially. Exactly. Yeah, the HP thing really kind of struck me

Starting point is 01:01:44 because I just remember that as like a communication no-no for how information is presented. And that's actually a good segue, by the way, to the crypto section, because I want to ask you guys, and this is going to help me break some, you know, I love doing a good taxonomy of definitions in any podcast. Because one of the things we talk about in crypto is the ethos of decentralization.

Starting point is 01:02:03 Sometimes the information is public and a public blockchain. It's often open source, distributed. It can be real time. I don't know if it's necessarily accurate information, but the information can be corrected very quickly, which then makes it more likely to be accurate because of the speed of revision, which, by the way, we also saw in the recent election, I think, compared to media. One of the observations people made is that media didn't move fast enough to or even

Starting point is 01:02:26 want to because of biases, their polls and predictions, whereas the prediction markets were faster self-correcting. So one question I have for you guys to kind of kick off this section about the underlying technology and how it works is first, let's tease apart all those words. I just gave you like a big buzzword, bingo soup of words. What are the words that actually matter when it comes to this context of eliciting better information and aggregating that information in a market? Like what is the key qualities that we should start with? And then we can talk about the technology is underlying that. One way of answering that question might be like the largest prediction market was the polymarket crypto prediction market. And the question is, is crypto

Starting point is 01:03:08 a necessary part of this? And I think the answer is probably no. I think why was the crypto market particularly successful? Well, because it was open to anybody in the world barring US citizens, right? Yes. And the market, because of that, was much thicker than the other markets. So there are some prediction markets which limit people's best to $1,000. And the crypto whale was betting millions of dollars on these markets. So that's why the crypto market, I think as a kind of regulatory arbitrage, became very important. And now the FBI is kind of looking at this. The French are looking at this.

Starting point is 01:03:44 Was it legal? Is it violating some laws? But I think the crypto part of it was not actually necessary. Yeah. I'm glad you pointed that out too. Alex, because I think people have been kind of hyping and over-inflating the crypto part of it. And I actually agree with you completely. Like, I don't know if crypto was at the heart of the way that that market works, except in those

Starting point is 01:04:04 qualities you mentioned. Scott, any thoughts on that point? So I totally agree with all of that. One thing that crypto does very well on top of being open and interoperable and transparent is it enables commitment, right? You can write a piece of software that is going to run in the exact specified way. It can be audited by all of the users. and then they can be convinced that it's going to run correctly.

Starting point is 01:04:25 And some ways we do information elicitation have challenges with commitment. If you're going to survey people and pay them six months from now based on whether their survey estimate was accurate or not, they might be worried that you're not going to show up and pay them. And so long as whatever the information is can also exist on chain, right? The resolution of the uncertainty can somehow be visible on chain

Starting point is 01:04:47 either through an Oracle or if it were like an on-chain function to begin with, like just what is the price of this asset or something, you can commit in a way that you can't necessarily or you can't do easily without complicated contracts. You can just commit that it's going to run as expected. Now, in order for that to work, your informationalicitation mechanism has to be fairly robustly committed and often also decentralized.

Starting point is 01:05:11 Like, Polymarket, by contrast, famously changed the terms of a couple of their resolutions because something happened that didn't quite make sense in the context of the way they'd said they were going to evaluate the outcome. And so they post hoc, this is after people have already bought in under the original terms of resolution, change the terms of resolution. And so that's like a lack of commitment that's actually, you know, hard for markets to like form when people don't trust that they're going to be resolved as described. Right. I mean, isn't it the most basic rule of markets? Like, you can't just suddenly change

Starting point is 01:05:42 the rules under you. Isn't that way we always talk about why we don't trust governments that don't enforce property rights and whatnot? Like, you just can't mess around. No, you're exactly right. And in the same way that blockchains create a form of property right that you can trust even without sort of a very trustworthy entity having established it, because the property right itself lives in this immutable ledger. Same thing here. Like you can, at least in principle, set up resolution contracts that are trustable and immutable and therefore expand the scope of the set of marketplaces we can configure, right? Yeah.

Starting point is 01:06:15 You know, it's not just the set of tools we had when you have to be able to trust the market organizer, but actually now this sort of like, you know, commitment enables you to go further. Just to break this down a little bit more, because I think you said some really important things in there. And I want to pause and make sure we flesh it out for our audience. So first of all, based on what Alex said earlier in the case of Polly Market, one of the key points was public and the information being out there. That's one. I mentioned earlier the example of it being updated quickly, as compared to media at least. You just mentioned the importance of credible commitments, and we've often described blockchains as a technology

Starting point is 01:06:49 that blockchains are computers that make commitment. So that's a third or fourth. I don't know the number count, but I'll just keep releasing the features. And then you also mentioned potentially decentralized, but I couldn't tell if it really needed to be decentralized or not. Can you give me more bottom line on decentralization where you stand there? Yeah, it's a great question.

Starting point is 01:07:06 And actually, maybe we should have started here. The necessity of all of these different features moves around with the type of market. The more complicated your information elicitation mechanism is, and this is especially important for the context where sort of pure information markets don't work, the more complicated your information elicitation mechanism is, the more likely it is that you want something that looks like crypto rails.

Starting point is 01:07:25 Ah, that's actually good to know. Right. Okay. So, like, if Hewlett-Packard is running an internal prediction market, first of all, it doesn't have to be open to the entire world because you're only trying to learn information from your employees, right? So openness is important within the firm, right? Maybe there's someone in the mail room who knows something that you don't know they know, and so you actually want that market of people to be able to participate.

Starting point is 01:07:45 But Hewlett-Packard does not necessarily care what a person on the street thinks about printer sales and certainly doesn't need to build the architecture to bring in like random people's estimates of printer sales, right? And so, you know, you need some amount of transparency because you need people to be able to see what the current price is and, like, see whether they agree or disagree and they can sort of move the price around.

Starting point is 01:08:03 But in other types of elicitation mechanisms, maybe you don't need transparency, right? If you're just going to pay someone based on the accuracy of their forecast down the line, you don't need them to be able to see what else is happening. You just need them to believe that you have committed and that the final accuracy is going to be transparent, that they can verify that you didn't just stiff them by the thing they predicted

Starting point is 01:08:23 happened exactly, but then you said, no, it didn't, and then you don't pay them. And so transparency is important only there with respect to the resolution, not with respect to the interim states. Yes, yes. But by contrast, like commitment is incredibly essential and needs to be believed or else the user won't even participate. Right. By the way, great that you gave the example of the transparency. And I'll let you finish your example in a second,

Starting point is 01:08:44 But I'm just jumping in because it reminds me of how we talk about the things that can be done on-chain and off-chain when it comes to scaling blockchains. And like, Provers versus verifiers when it comes to zero knowledge or whatnot. And it's really interesting you pointed that out because I want to make sure people who are listening who are builders listen to that because that means you can do certain things on-chain in order to whatever your goals of the design are and then put other things off-chain. Like you don't have to have this purest view of how truth must be transparent. It's very smart to point that out. Anyway, keep going with your other example. Yeah, and I completely agree, by the way. I mean, like, one of the things, when I talk to teams, I'm constantly trying to get them

Starting point is 01:09:21 to think about which features of the marketplace are the most essential for market function. And it varies by market context. And even if eventually you're planning and having all of these features, right? Like, as you're deciding, like, which thing do we build first or, like, as we're progressively decentralizing, like, what do we prioritize? You actually have to understand the market context you're working in. That's so smart because it's basically not. other way to hit product market fit too, because then you're not like overbuilding and over featuring

Starting point is 01:09:48 something. Anyway, yeah, but keep going with your other side of that. Totally, 100%. So to get to the question of like, when does decentralization matter? Decentralization has lots of different components that might make it matter. One of them is just the ability to like make these commitments even more enforceable. Like it makes it possible to be confident and function and liveliness and so forth. All of those things are important for a market because if your prediction market goes down, the night before the election, you know, first of all, you lose the information signal from it. Second of all, you lose the ability for people to participate in the market, which would sort of adjust the price and move the signal around. Similarly, if you lose the ability to, like, resolve the

Starting point is 01:10:26 truth, then maybe you can't finally resolve the market and you have all of these bets that are sitting in limbo because the market doesn't know what happened. The key is everyone is bringing in their own information, but in order to finally resolve the contract and determine who gets the payout for the bet, you have to have the chain have a way to know what actually happened. Another place decentralization is sometimes very important is in that resolution function. Like, you know, if the market is on chain, you somehow have to get what actually happened onto the chain. And maybe the biggest better happens to also control the one resolution function. And so they can now sort of rob the prediction market by just lying about the resolution of the event.

Starting point is 01:11:04 They tell the system like, you know, candidate A1 when actually candidate B one. And then by the time people realized that this wasn't correct, they might not have a way to fix it. But even if so, that person might just be gone. So decentralization and resolution, just like we think about decentralized Oracle sort of mechanisms, this is basically an Oracle, right? You have to bring off-chain information on-chain in a lot of these contexts to resolve the contract, or if you're doing this in a centralized platform, the users have to trust the centralized platform to resolve the contract correctly. By contrast, if the information does not need to be brought in through an Oracle, right,

Starting point is 01:11:36 if it already lives in a system that's verified and the resolution is like provably going to do what's claimed, then you don't actually care about decentralization, say, in the discovery of the resolution, you're actually just like reading information and your commitment contract takes care of everything else. And just really quick, Scott, you've said Oracle a few times. Can you actually properly define what you mean by Oracle in this context? I know we talk about a lot in crypto. Yeah, and indeed, Oracle is not a completely uniformly well-defined term. In this context, I'm talking about oracles as like a truthful source of information about what the actual resolution of the event was. So if Trump won the election, the Oracle tells us Trump won the

Starting point is 01:12:12 election. And if Harris won the election, the Oracle tells us Harris won the election. And the reason we're using that is because the election is, of course, not being conducted, or at least maybe in the future, we can dream. But in 2024, the U.S. presidential election was very much not conducted on a blockchain. And so if you're going to have an on-chain prediction market, you somehow need the chain to be able to learn the information of what actually happened in the off-chain election. And so the Oracle is like basically the source of that information. The key of the Oracle, as God say, is to bring it off-chain and bring it on-chain. I mean, the thing about off-chain is that people can look at the New York Times, right?

Starting point is 01:12:51 And so the New York Times is often considered a Oracle in that you go by whatever is printed in the New York Times. That would be a way of resolving a lot of bets. Like, did the New York Times report the Trump won? That might be one way of resolving these bets. Yeah, great. But the key problem is to bring that. off-chain knowledge on-chain in a way in which the information is not distorted in the transmission. And the reason why that transmission, you're worried about it being distorted,

Starting point is 01:13:20 is precisely because it's the revelation where all the money is, right? Right. So there are big incentives to distort the transmission of that information. In fact, a lot of the crypto hacks, which have happened, have happened because people found a way of distorting the Oracle and then using that on the crypto market. So, you know, the market resolved in one way. And if you can change the Oracle, then you can make a huge amount of profit out of doing that. So there's a big incentive to mess with the Oracle. That's why it's really difficult. And we can stick with the New York Times example, right? A lot of people are going to make their morning trading decisions based on what they see in the New York Times and on the Bloomberg terminal and so forth. And so if you could, in a

Starting point is 01:14:04 coordinated way feed the wrong information to that, it would change many, many people's behavior, and you could trade against that because you knew that they were going to get the wrong information. Exactly. So this can happen in the off-chain world, and indeed we saw there was one tweet, right, that the SEC is going to legalize, you know, ETF Bitcoin contracts. It looked like, you know, it was an official ruling, and it turned out to be a hack, turned out to be correct, but that wasn't revealed until days later. But, yeah, so if you can distort an Oracle, you can make money. Totally. Or, I mean, if we're talking about the New York Times, it would be remiss for us to not have the, like, Dewey Defeats Truman, right? You know, famous, famous front page, like, huge text headline that just

Starting point is 01:14:44 turns out to be inaccurate. Right. That's a famous case of what we did in media at Wired, too. It's called the pre-write, and then you accidentally print it sooner and you get it wrong. There actually have been cases of someone. You write their obituary months or years in advance, and it goes out and says they're dead. Okay, you conflated earlier, and I agree they're generally connected and similar, but there are some nuances between decentralized and distributed. Like, distributed can just be like redundant systems that have multiple, like the system going down is what you were giving the example the night before something. That's a case where being distributed matters, but it doesn't have to be decentralized necessarily. Correct.

Starting point is 01:15:17 Like, i.e., there could be distributed nodes managed by a centralized entity, for instance. Absolutely. So I want to make sure we're very clear about the distinction between decentralized and distributed as well. Totally. Whereas by contrast with the oracles, for example, you might really care about being decentralized, right? You might care that no individual entity can sort of unilaterally change how the contract's resolve. Exactly. Just one other point, another advantage of doing all this stuff on blockchains is that it's composable.

Starting point is 01:15:41 It's not that we're just like intrinsically interested in some of these questions. Like maybe so, right? Some people are just like, you know, intellectually curious, like who's going to win the presidency in a month. But rather, like, lots of other stuff depends on it, right? If you're making decisions about which supplies to order in advance, you need to have beliefs about the likelihood that tariffs are imposed, under the next administration. And so having these things live on open composable architectures is useful because they can be wrapped with other information and other processes.

Starting point is 01:16:11 You can tie your corporate operations in a very direct way into these sort of information aggregation mechanism signals. Yeah, to put it even a more basic way, just because I don't know if everyone necessarily knows composable in the way that we talk about it, it's like the Lego building blocks, the markets on chain, or the information on chain, is. is a platform that people can build around, build with, bring in pieces of information, combine it with other tools, et cetera, and you can create, like, different things. And that's a composability.

Starting point is 01:16:39 And I'll put a link in the show notes to a post explaining composability as well. And then the other quick one is open source. Does the code itself have to be open source, auditable, public good? Again, it depends how much you trust the market creator. Yeah. And again, this is true across the board for applications that can be run on blockchains or not. You're always making tradeoffs between, you know, trust through reputational incentives and institutions and trust through code. You know, for example, like in actual commodities markets, there's a lot of trust through institution and legal contract,

Starting point is 01:17:14 but there's an architecture in place to establish the trust between the institutions and the contracts and their enforceability via the institutions for those contracts to be real enough that people believe in them enough to pay money for them and to have all of these market features. blockchains enable these sorts of trusted activities in lots of contexts where the institutions are not strong enough or present enough to do it for you. If you're having like $5 bets, like small money bets on some incredibly minor question, like will the horse that wins the Kentucky Derby have a prime number of letters in their name or something like this, right? You're not going to have necessarily an institution that is even able to evaluate and like set up that contract in a way that is worth.

Starting point is 01:17:58 doing at the amount of money it's going to raise. I like how Scott changes that Kentucky Derby into something he would be interested in. Oh, well, if it involves prime numbers. Horses, forget horses, but prime numbers. That's so funny. I love how well you know. I will have you know. The Kentucky Derby is also interesting because it has all sorts of cool statistical questions

Starting point is 01:18:20 going on. And cool hats. Fascinating hats. Absolutely fascinating hats, pun definitely intended. I love it. So, like, substituting code for the source of trust for these, like, very unusual or sort of, like, micro or international, there's not a clear jurisdiction, right? All of these contexts sort of push you more into security via code rather than security via institution. Let me add one more point on the blockchain.

Starting point is 01:18:44 So I think generally speaking, as I said, the blockchain is not necessary. However, as we're looking towards the future, it may become more and more useful to have these, you know, very decentralized rails. So Vitalik Boutteran recently wrote a post on Info Finance talking about prediction markets. And he credited you at the top as one of the people reviewed it. But yeah, keep going. Exactly. And so one of their interesting points which he made is that AIs may become very prominent predictors. They may become very prominent participants in these prediction markets.

Starting point is 01:19:19 Because if you can have a lot of AIs trying to predict things, well, that lowers the cost tremendously. and that opens up the space of possibilities of what you can use prediction markets for. And so the blockchain is very good for, you know, nobody knows you're an AI on the blockchain. Right, right, right. And so if we're going to have a lot of AIs interacting and acting as participants in markets, then the blockchain is very good for that. That's absolutely right. And we have a lot of content that's already on this topic, which actually gets at the intersection of crypto

Starting point is 01:19:52 and AI and where they're a match made in heaven, in fact. not only because of AI centralizing tendencies and crypto's decentralized tendencies, but because of concepts like proof of personhood, being able to, in privacy preserving ways, yet even if it on a public blockchain, find ways of adding attribution. And there's just so much more that you can do with crypto. I agree, Alex, and I'm so glad you brought that up. It's funny because when you were saying earlier in the early definition of a prediction market as this way to kind of elicit information that's dispersed across many people,

Starting point is 01:20:24 I immediately went to like, oh, that's the original AGI. If you think about artificial intelligence, let's just talk about human intelligence at scale. Like, that's what a prediction market can be. I do want to make sure we also touch on other applications, a little bit on the future. One quick thing, though, before we do that. So now we've summarized some of the key features. We've talked about the election. We've talked about some of the underlying market foundations and some of the nuances.

Starting point is 01:20:46 We've talked about what does and doesn't make prediction markets work. And also mentioned earlier that they're part of a class of medicine. mechanisms that can aggregate information. So I want to really quickly, before we talk about applications in the future, near future, I want to quickly summarize what are some of those other mechanisms that could get at this kind of information aggregation that aren't necessarily prediction markets. Awesome. So first of all, like, again, just to think about what is this class of information aggregation mechanisms. And Alex defined it earlier, it's these are mechanisms that bring together lots of dispersed information to produce like an aggregate statistic or set of

Starting point is 01:21:23 statistics that combine the information of many different sources, and ideally that that aggregate is informative. Now, there are lots of ways to do that, right? Some of the simplest ones we actually talked about earlier are just to like ask people for their predictions and later pay them based on whether they're correct, right? And you can do that with random people, wisdom of the crowd style, or you could do that with experts, right? And so like very simple types of information aggregation back is they're just like incentivize people to tell you what they know. Or even just go and survey them, right, surveying people like in an un-incentivized context, but where people have no incentive to lie and just, like, have an opinion, right? They don't have to do any research or, like,

Starting point is 01:22:00 invest in any effort to know their version of the answer. You just run a survey. But then, you know, sort of there's this whole menagerie, maybe, of incentivized elicitation mechanisms that are designed around different elicitation challenges. So I mentioned earlier pure prediction mechanisms. These are the mechanisms where you ask people for their beliefs and their beliefs about other people's beliefs, and then you use people's estimate of the population beliefs to infer, like, whether they were lying to you about what they believe and or, like, how informed they were in aggregate. So if you can use that to figure out where the person fits in the distribution, and pure

Starting point is 01:22:34 prediction is like an incentivized version of that, right? So you're going to actually pay people based on how accurate they are, but you're not paying them based on how accurate they are about what actually happens in the future. Rather, you're paying them based on, you know, how accurate they are about the population estimate. Right. And so that enables you to pay people up front. immediately. These are used for like, you know, subjective information or sort of like

Starting point is 01:22:55 information that's dispersed among small populations. Maybe it's not big enough to have a thick prediction market, but people are informed enough that if you can directly incentivize them to tell you the truth, then you can actually like aggregate the information usefully. A couple of my colleagues at HBS, Reshma Hassam, Natalia Regal, and Ben Roth have this beautiful paper where they use these peer prediction mechanisms in the field in developing country context where they ask people who in their community is likely to be the most successful micro entrepreneur. And then they allocate, you know, sort of funding according to these predictions. And it turns out that like the predictions are actually quite accurate.

Starting point is 01:23:29 So like the incentivized peer prediction mechanism sort of produces answers that line up with like who actually ends up being successful in these businesses down the line in a way that is more effective, say, than just asking people and telling them, oh, we're going to allocate the money according to whatever you said because then people will lie and say, oh, my neighbor or my friend is like, you know, the best. I'll put that paper in the show notes too. Yeah, it's a great paper. Super fun to read, very readable, too.

Starting point is 01:23:52 So one way in which the wisdom of the crowds doesn't work, of course, is when the crowd thinks they know the answer to a problem, but they actually don't. Oh, okay, of course, yeah. So there's this great paper by Freilac and Song and McCoy, and they give the example of, suppose you ask people, what's the capital of Pennsylvania? And most people will think, oh, well, it's probably Philadelphia, right? It's the biggest city, popular city, you know, American Heritage, Liberty Bell, all that kind of stuff.

Starting point is 01:24:21 But it actually is the wrong answer. So if you go just by the wisdom of the crowds, you're going to get Philadelphia, and that's wrong. The correct answer is actually Harrisburg, which most people don't know. However, a small minority of people do know the correct answer. So how do you elicit this? So their mechanism for doing this is what they call the surprisingly popular mechanism. And what you do is you do what Scott says is you ask people, not only what do they think is the correct answer, but what do they think other people will say? And most people, of course, will think, well, I think the correct answer is Philadelphia. Other people will say Philadelphia. But then you're going to see a bump, right, of Harrisburg. It's going to be very surprising.

Starting point is 01:25:05 It'll be a substantial number of people will say Harrisburg. And that will be quite different than what people expect. And if you choose that, the authors show that this can improve on the wisdom of the crowds. So the surprisingly popular answer, the answer which a minority chooses, in contrast to the majority, that can actually get you more information out. So depending upon the question, there are these clever ways of pulling this inco-hate information out of the crowd and eliciting the truth, even when most people in the crowd don't know the truth. That's fantastic. I'm obviously going to include all these things we're referencing in our show notes, but that one is really interesting.

Starting point is 01:25:48 Right. That's wild. And then maybe one other piece in the menagerie, of course, the listeners of this podcast will be very familiar with are simple auctions, right? Auctions are information aggregation mechanisms too. We talk about price discovery in an ordinary, like, sort of very liquid market as being an information aggregation source. But some markets aren't like big and liquid all the time. They don't have like lots of flow transactions. Maybe it's a super. rare a piece of art, but an auction is still exactly useful for figuring out what the art is sort of like worth in the eyes of the market. And you can often discover things, right? Like there's some artist that was not popular to the best of your knowledge and then they have a piece with like a major sale and people's estimates of the values of all of their other works change accordingly because of the information that's been revealed about people's change in taste or whatever from this one sale. While we're thinking things for the show notes, there's an incredible book. Oh, actually, I think this is

Starting point is 01:26:40 my very first A16Z crypto booklist contribution called Auctions, the Social Construction of Value by Charles Smith, which talks about auctions from a sociological perspective as a way of establishing and understanding of value in a bunch of different contexts. That's great. And by the way, I do want to plug the episode you, me, and Tim Roughgard and did where we literally dug into auction design for Web 3 for hours. That was so much fun. So as we've been like arching through these different types of mechanisms, it's a really good reminder that the type of question you you're asking the type of market participants you have. And like this, we were just saying it shapes your decisions about how to, like,

Starting point is 01:27:17 structure your market mechanism. It also shapes your decisions about what type of market mechanism to use, right? Like, if you think that the population is not super informed on average, but like informed at the second order level, then this mechanism Alex was describing is like perfect. Because the information's there, it's just not like immediately apparently there. Right. What I love that you guys are talking about and we can now segue into some quick discussion of some applications in the future, and then we can wrap up. We've been talking about implications

Starting point is 01:27:44 for design throughout this podcast, but I think it is very interesting because you've been saying throughout, both of you, that it really depends on the context and your goals and then you can design accordingly. And that's actually what incentive mechanism design is all about, as I've learned from you and Tim Roughgarden and seen over and over and over again. But two quick things, just lightning around style that I want to make sure I touch on. One, multiple times you both have alluded to this payout feedback loop, like I'm inferring from what you've said, that the payouts have to be almost quick that you get like an instant feedback loop on your outcomes, because you gave an example earlier where if it's like delayed by two weeks or so-and-so, it may be less effective.

Starting point is 01:28:19 Is that necessarily true? Depends on trust and attention, right? Some people have said that one of their concerns about prediction markets is that people like betting on sports because, you know, it's happening in real time. You know the answer within a couple of hours or in the case of a horse race within minutes, whereas these prediction markets often take months to resolve the final answer or the time of resolution might not even be known, right? It might be, you know, sort of who will be appointed to this position. So there's possibility that speed is relevant for who chooses to participate in some context, whether they find it fun. The other context we were talking about is when time matters for

Starting point is 01:28:52 trust. If you're in the developing world trying to figure out how to allocate grants, people might not trust or even just have the infrastructure support to participate in a mechanism where they're going to be paid six months out based on the resolution of some confusing outcome. Whereas if you can pay them today, they'll participate today. hence why they experimented with peer prediction mechanisms in that context in the first place. It was sort of a setting where you could in principle pay people based on the outcome, like how successful their neighbor was being an entrepreneur with whatever grant they'd received. But a lot of complexity goes into actually doing that in practice because you have to track down the people again and all of that.

Starting point is 01:29:28 Ah, yeah. One other quick buildery thing that came up, again, seems so obvious to you guys probably. But the best systems are where their prediction markets and such systems work when there is a discreet. event, like an election or something to be resolved, it probably wouldn't work for some ongoing, kind of loosely defined, non-discreet event? So the prediction market mechanism, sort of like the canonical prediction market, as we've described it, is a mechanism where you're buying like an asset that has a payout as a function of a discrete event.

Starting point is 01:30:01 But that is, of course, not even the average case of markets, right? Like, you know, when you're buying oil futures or something, most of the transactions in many of these markets are actually sort of in the interim. It's based on changes in people's estimates. And so if you have a market where, you know, it's possible to sort of continually update and trade, you know, as estimates change, then, like, you can still gather a lot of information, even if the value attained is in a flow or, you know, in stages or something of the sort. It doesn't have to be sort of a single cutoff date. I think you can design them in different ways.

Starting point is 01:30:33 They do have to resolve at a point in time, but the way that they resolved could be based upon a stock price or something like that. Yeah. And you can have like dividends or something, right, too. You can have things that pay out over time based on sort of interim steps. Like, you know, lots of things have continuous payouts based on like the growth of a company or something of the sort. And so you could imagine like prediction securities that are kind of like that. I.e. the stock market. Exactly. I e. the stock market.

Starting point is 01:30:58 Right. The HP example I gave earlier divided the time into two-month period. Right? So is it May to June or is it July to August? Is it September? October. So, you know, you can always take a continuous event and chunk it into five or six discrete periods. Yeah, yeah, even if somewhat arbitrary. That makes so much sense. So so far, these prediction markets have been used just for what we've been saying for predicting something. But you can also create, and here I'm going to riff off Robin Hanson again, my colleague on these questions. and he says we can also create these conditional markets. So the question would be something like, as I said earlier with Futarky,

Starting point is 01:31:39 what would happen to GDP if we put together this science policy? Now, we might not want to jump all the way from democracy into Futarky in one go. We're probably not ready for that. We're not ready for the full hands. Not quite ready for prime time, I think. Yeah. But here's a fascinating idea of Robbins. which I think we are ready for, which we should use.

Starting point is 01:32:04 And that is, what would happen if we fired the CEO? So this is a huge question that companies want to know. You know, we saw a few years ago, it was kind of remarkable when Steve Bomber left Microsoft and the stock price went way up, you know, suggesting that the market thought that Bomber was not a great CEO. Or we just saw, you know, with Brian Nichols, he moved to Starbucks from, five guys. He'd be extremely successful at five guys who moved to Starbucks on the day that Starbucks announced that they were hiring Ryan Nichols as CEO, the price of Starbucks jumped up.

Starting point is 01:32:44 So why, however, do we need to wait? How about creating a continuous market, which says at any given time, would the price of Starbucks be higher if they fired the CEO? And so you can create these decision markets, prediction markets, you create a prediction market in, would the stock price be higher if we had the same CEO, or would the stock price be higher if we fired the CEO? Now, that's an incredibly useful piece of information. Companies, this is billions of dollars every single day, are based upon exactly this question. And that's a question which I think decision markets, prediction markets, would be really good at answering.

Starting point is 01:33:27 we already have the stock market. People are already investing billions of dollars in exactly this question and we can make it more precise and more detailed and more usable. What I really like about that application is it leverages a type of information that people are already developing, right?

Starting point is 01:33:44 People are spending a lot of time reasoning about what's going to change the stock price of Starbucks and they have a lot of different refined ways of doing it, but it uses it to address a question that's like useful sort of as a practical hypothetical. As Alex said, it brings the information forward in time. You know, normally in a current market context, we can only learn what happens if Starbucks

Starting point is 01:34:05 replaces the CEO when they replace the CEO. But actually, that's like the least important time for us to learn that. We actually want to know it, like, when they're deciding, should they replace the CEO? Yeah, exactly. You want to know it before. Yeah. And so being able to harness that same effort that people are putting into understanding what affects the stock price of Starbucks and like, you know, which companies are well run and which aren't, And pushing it towards this question can reveal important information at a time when it's more useful leveraging things people are already good at predicting. Exactly.

Starting point is 01:34:36 That's such an interesting and such a useful and extremely real and possible right now thing to do. We're not just being crazy futuristic like 10, 15, 20 years from now. That's so great. Can I be crazy futuristic? Put you in a little bit more. Yeah, yeah. We actually want a little of that. Go for it.

Starting point is 01:34:51 You're absolutely right. The should we fire the CEO market could be implemented, right now and it would be extremely useful. And it's the first step towards making more decisions by like Dow's, by a blockchain consensus, right? I mean, so if you can make a decision about should we fire the CEO, should we expand into Argentina or into China? Should we have a new model this year? Right. You can start asking the market lots of different types of these types of questions. So let's start with should we fire the CEO, one of the biggest and most important, most salient of these questions, whereas Scott says it's an information-rich environment. People are already collecting lots of information on exactly this question.

Starting point is 01:35:35 And once we've got some experience in this market, we can start applying it to further markets down the line. Footnote, okay, I love that application too. And that ties into the importance. We talked earlier about, you know, maybe running these markets in like an internal currency. You know, an advantage there is you can use it to put everyone on the same footing at the outset, right? Like, you know, the Starbucks CEO question, there are many different sort of like very high value and high ability to trade entities that already are like participating in this style of question. Whereas for a Dow, you actually might have tremendous inequality and wealth of the participants, but you can make them wealthy in proportion to their reputation or something, you know, in the internal token, which can then be used to like, you know, sort of have them all participate equitably at the entrance to these decisions. I love this. And this is where I'm very proud that we have published a deep body of research across many people, not just our own team, into DAOs, what makes them work, what doesn't work, what's effective governance mechanisms. I'm going to link to that in the show notes. Because also we're arguing that sometimes you can do a lot of these things, not just in the crypto world, but you can apply them to other decentralized communities. And I want people to remember that that's a useful use of DAWS, which are just decentralized autonomous organizations. Are there any other pet applications, either current or futuristic that either of you have? I have one, but I'm going to wait to wait to.

Starting point is 01:36:49 you guys are done. I mean, two other very quick hits. You know, we haven't touched directly yet in the podcast on the idea of markets for private data, right? For like, you know, another form of information aggregation is, you know, maybe a lot of people have information that will be useful in designing a new pharmaceutical or medical treatment. And they have their own private information of this form. And we'd like to be able to elicit from them in a way that also fairly compensates them for their participation or something of the sword. And we have some mechanisms for this already. Like you might have surveys managed by a health center and they pay you sort of a show up fee for participating in the survey or whatever. But there's a possibility for much richer markets of that form that leverage

Starting point is 01:37:33 sort of like individual data ownership and like permissioning and so forth. Yeah. One example, by the way, just concretely is like in the DCI movement, decentralized science where people are putting their information like medical data using blockchains to bring more ownership, transparency, consent, which they don't have. That's just one example. What's the other one you had, Scott? The other one, you know, is getting incentivized subjective beliefs, right? We've talked a lot about predictions of things that are, have an objective truth. But another big frontier for information aggregation is getting really good estimates of things that people believe that are fundamentally subjective, right? And like, you know, if you're trying to do like market research for

Starting point is 01:38:14 your product, you know, do people want this? You know, one of the advantages of crowd. For example, is that it's a better information elicitation mechanism, or you could go and ask 10,000 people, do you want to buy this? And some of them might say yes. But unless you're actually taking money from them, you don't know whether that's like a truthful representation. Yeah. And so crowdfunding lets you learn about the total market for your sort of initial version of the product in a way that's incentivized. More broadly, I think like subjective elicitation is like a really important direction to go into. Can you quickly maybe give a very short definition in the uniquely crypto blockchain context of a Bayesian Truth Serum here? Because isn't this where Bayesian Truth Serum apply? Sure. I mean, the Bayesian Truth Serum is actually an example of those pure prediction mechanisms we described. And there are many different versions of it. But loosely, the idea is if I ask you your opinion on something, did you like this movie? And then I ask you, what's the likely that, you know, another person I ask will say that they liked the movie.

Starting point is 01:39:08 You might have a reason to lie to me about whether you like the movie or not. You might say, oh, I really liked it because, you know, you produced it. What am I going to do? But you actually hated it. your estimates of everybody else's beliefs will be sort of tilted in the direction of them mostly disliking it. So long as I'm going to reward you to proportional to your accuracy.

Starting point is 01:39:25 You know that you disliked it and so everyone else probably will too because you're abasian. And so I can detect looking at everybody else's responses, I can detect whether you sort of like told me a distribution of other people's beliefs

Starting point is 01:39:38 that's consistent with what you said your belief is. Great. One of my quick applications and kind of an obvious one but I want to just call it out because I find it very boring when people say the same thing. Like, oh, media, da, da, da, whatever. What I find very interesting is, and people often talk a lot about having mechanisms for,

Starting point is 01:39:54 quote, finding truth. But sometimes I find it to be very pedantic and moralistic and equally as grading as a way that the very people they're trying to bring down. And so it's a pet peeve of mine when I'm on the Twitter discourse, like, oh, God, I'm so bored by this. But I do find it very interesting that some of the commentary surface at prediction markets for basically resolving more accurately and faster than mainstream media, but not having some of the same filtering of partisan interests. I mean, although this might be different with certain communities of DAOs,

Starting point is 01:40:21 if you do predictions limited to certain DAOs. Yeah, again, it depends who's in your market. Yeah, exactly. This goes back to your point about thick and thin. But it's also interesting because it's a way to put a little bit more skin in the game, which is one of the biggest drawbacks in current media, is like the people writing don't have skin in the game, which is why I've always been a believer in not having third-party voices,

Starting point is 01:40:39 but the experts write their own posts and then editing them. is more interesting to me. So I do think it's very interesting to think about this use case of reinventing news media using prediction markets. And Vitolix Post actually had a great headline, which is that think of a prediction market as a betting site for participants and a news site for everyone else. That'd be my application. So I think more generally, it is odd how we do quite a bit of journalism. So for example, it's totally standard practice for a financial journalist, right, for it to be against company policy, for them to invest in the companies which they're recommending, right? And as an economist, I kind of think, wait a second, don't want the exact opposite, right?

Starting point is 01:41:23 Yeah, you want more skin in the game, exactly. Yeah, more skin in the game, right? So, you know, I say that a bet is a tax on bullshit, right? I like that line. That's a great line. I love it. So, you know, how about you have to be upfront about it. You have to be honest about it, transparent about it. But maybe journalists should say, this is what I think will happen. And these are the bets, which I've made. And you can see my bets on chain, right? Yeah, yeah.

Starting point is 01:41:46 And let's see what their past track record is, right? Like, it's kind of amazing that we do not have any track record of opinion editorialists whatsoever. Only Tetlock, you know, started to create that and found that they were terrible, right? But how about let's create a series of bets and on chain? And this would, you know, change the types of things. people who become, you know, editorialists who get these jobs in the first place, right? So let's start making sure you bet your beliefs and then let's promote people whose bets turn out to be accurate. And that's going to change journalism entirely if we were to change

Starting point is 01:42:24 the metrics by which journalists are evaluated. I agree. Annie Duke talks a lot about this too. Yes. It's not just bets like in a binary true false way, but bets that are weighted in terms of likelihood, probability of accurate. Like, you don't have to make a binary, like, it will be this or that. Absolutely. I believe 80% that X will happen. And that is also another way to kind assess in a more nuanced way. And that gives a lot of room for the nuances that are often true when it comes to guessing the truth. Absolutely. Exactly. There's a big incentive to say, this is never going to happen. This is impossible, right? But then if you ask them, well, if it's never going to happen, are you willing to bet $10 that it might happen? And of course,

Starting point is 01:43:04 They should all be willing to. Of course, they're never willing to make those bets. That's right. Even people who hate Elon Musk as journalists will then start saying, well, actually, I'm going to bet on that guy for building X to happen because I saw that, you know, shuttle launch. And now I'm thinking, okay, maybe I'll increase that from 10 to 20 percent or whatever. Yeah. Exactly. So betting could reduce the hyperbole. Yeah, it's exactly right. Yeah, totally. By the way, this ordered on some other really critical information elicitation mechanism that uses a different version of this sort of cross-examining some people's belief. against others. If you think about community notes on Twitter, that's an information aggregation mechanism, right?

Starting point is 01:43:39 It's like getting a lot of people's opinions and then only deciding that they're correct if you have agreement from people who usually disagree. Yes, exactly. Because that's where Wikipedia failed when they had the cabal of expert reviewers. They didn't have that kind of check and balance mechanism. Yeah, totally.

Starting point is 01:43:57 Yeah, community notes is a great one. I have one last question for you guys because we don't have enough time to go into policy. You know, in general, like some of these became popular because they're offering contracts that were banned from the market. So a big question is whether the offshore crypto markets will follow the rules or not. So how do you sort of create innovation, obviously, in that environment? To me, the core question here is what's the difference between gambling and speculation? Is there a difference? I'm curious if you guys have a thought

Starting point is 01:44:21 on as a parting note on this. I mean, so one very important thing to remember is that depending on the context, like you may be in a different point on a continuum, right? Like part of what What makes sporting events, like, exciting and suspenseful is that there's a lot of stochasticity and, like, you know, sort of the amount of information that any individual has is reasonably small, even if they put a lot of effort into figuring it out. But there might be some amount of, like, you know, sort of informed betting in sporting events. And then as you move towards things where there's a lot of information to be had and a lot of, like, value also to knowing the answer, right, a lot of market value to actually figuring it out, right? Like, how do we

Starting point is 01:45:00 allocate goods in markets, right? Going back to the very beginning when we're You were talking about the role of markets and determining the value of something and clearing supply and demand, right? Like there, there is value generated through the process of people engaging. Now, there's one really important caveat about speculation. We talk about this like a lot in Cryptoland, right? There is speculation of the form. I have beliefs and, you know, I'm investing to support a product that I think will exist and that I want to exist and that I think other people will want. And then there's also speculation on speculation, where you're actually not so much betting

Starting point is 01:45:35 based on your own beliefs. You're betting on what you think other people will choose to bet on. Like we talked earlier about hurting, you know, you might place bets because you think other people are going to place bets in a given direction, not because you actually have any information about what's going to happen just because you have information about how the market might move. That's right. That's speculating on speculation. Exactly.

Starting point is 01:45:53 That's speculating on speculation. So there's this sort of like valuable type of speculation, which is people moving resources around in a way that reflects their beliefs and sort of like can help us make markets work better and achieve better outcomes. Like that's sort of in this midspace between the randomness where moving the money around has no impact on outcomes, right? You're just betting on coin flips, like, you know, your money does nothing. And this other edge where moving the money around becomes sort of its own project that is independent of outcomes. And so again, like sort of doesn't provide information. Right. Like these prediction markets are particularly well architected, again,

Starting point is 01:46:29 at least in the cases where they're very large and thick and all the things we've talked about that you need to make them work, they're particularly well architected to try and be in that midspace where the information provided is valuable and comes out of like real knowledge and activity in a way that actually sort of means the market does something valuable. Yeah, and by the way, on the earlier example, when we talk about a lot, the obvious examples where it plays out is like the Carlotta Perez framework of like speculation phase followed by an installation phase as like a driver of technology cycles. There's also the example, Bern Hobart wrote a piece for me a few years ago on how bubbles are actually a good thing when they have a certain type of quality in this case.

Starting point is 01:47:08 And he also wrote a new book about it recently for Stripe Press with Tobias Huber, which they go into greater detail about that. Well, I should read that. It's basically an example of quote, I don't want to put moralistic terms on it necessarily, but useful speculation that kind of leads to other things as an outcome versus speculating for the sake of speculating, which is partly the distinction you're pointing out. Well, I think people in Las Vegas who are at the slot machines, they're gambling because they have no way of influencing or of improving their predictions of what the slot machine is going to show up, right? It's just pure random chance. On the other hand, there are many, many areas in which we are trying to predict the future and in which investing can help us improve our predictions. And this is why I think prediction markets should be completely legal. should be legalized because of all the forms of gambling, of all the forms of speculation,

Starting point is 01:48:03 this is one of the most useful forms. So we want to incentivize the type of speculation or gambling, which as a side product produces these useful public goods, which is trying to predict the future. This is incredibly important. You think about all of the questions that we have, you know, what is happening with climate change, which of these scientific predictions are accurate? who is the best candidate for the presidency? All of these questions we have, prediction markets can help us answer these questions in a way which is more objective, more accurate,

Starting point is 01:48:39 and more open to everyone. So I think the case for legalizing these is very, very strong. That's amazing. I'm going to give you the last word on that, Alex. You guys, thank you so much for joining this episode. That was so fun. Thanks, Anel. Thanks, Scott.

Starting point is 01:48:52 It's been fantastic being here. Thanks so much. Really fun conversation. and QED. Bye, QED. Thank you for listening to Web 3 with A6 and Z. You can find show notes with links to resources, books, or papers discussed, transcripts, and more at A6NZ Crypto.com.

Starting point is 01:49:14 This episode was produced and edited by Sonal Toxy. That's me. The episode was technically edited by our audio editor, Justin Golden. Credit also to Moonshot Design for the art and all thanks to support from A6 and Z Crypto. To follow more of our work and get updates, resources from us, and from others, be sure to subscribe to our Web3 weekly newsletter. You can find it on our website at A6NCrypto.com. Thank you for listening and for subscribing. Let's go.

The a16z Show - Prediction Markets and Beyond

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.