Investing Billions - E347: The $26B CIO Who Turned Superforecasting Into Alpha

Episode Date: April 14, 2026

How do you manage a $26 billion public fund while keeping every investment decision disciplined, every team member calibrated, and every partner accountable? In this episode, I sit down with Mark Steed, Chief Investment Officer of AZ Public Safety Personnel Retirement System, to explore how super forecasting and probabilistic thinking shape portfolio management. Mark shares how lessons from Dr. Phil Tetlock's the Good Judgment Project inform every investment decision, why intellectual humility and calibrated confidence drive better outcomes, and how simplifying portfolios into broad buckets creates flexibility and competition for capital. He also unpacks the role of co-investments, structural alpha, and first principles thinking in public markets.

How does super forecasting play into how you managed $26 billion today? This is a great question. And I love that we're starting with Dr. Phil Tetlock. So Phil Tetlock spent decades proving that, like most experts are terrible forecasters. Not because they're like dumb, like in any way. It's just that they're like most of us overconfident. And like they don't keep score. So and I know this firsthand, but back in 2010, Dr. Tetlock organized a team to compete in

what was really a four year forecasting competition for the Department of Defense. And I was on that team, so I was a guinea pig. And over like the subsequent four years, which I think went from 2010 to about 2014, grinding through geopolitical questions with like real scoring, I learned what we call calibration and what calibration thinking actually feels like. So he eventually published his findings in the book, Super Forecasting, which became a bestseller on Amazon, probably around 2016 or 2017. And I basically brought that methodology back to my investment team.

what does that mean for us right now? Every investment decision requires a very explicit probabilistic forecast. And let me explain what that means. It's not like I think this fund looks good, but it's, I believe there's a 55% chance. For example, this say manager will outperform the MSCI health care index by 400 basis points over the next five years or five years from the close of the fund. So it's specific. You have to have a probability forecast. You have to have a clear definition of success in a time horizon.

is in a confidence level. So the discipline isn't just about, you know, being right. It's about being calibrated. So when you say you're 90% confident, are you 90% right? And so it's about knowing the difference between what you know and what the information allows you to know and what you believe. So how do you track your team's investments internally based on a super forecasting methodology? We use the briar scores. And the briar scores are something that were championed by Phil Tetlock. And he has since commercialized his research into what's called the good judgment project. So the Breyer score is just a mathematical formula that ranges from zero to two. So lower score is better. So a zero briar score would be that you're 100% confident and 100% right.

So you're basically like an all-knowing kind of omniscient omniscient. And a two is that you're 100% confident all the time, but like 0% right. And just for reference sake, a briar score of 0.5 would be that you're sort of a coin flip. But it's not just about the briar score. You also have to attached kind of a rationale, you know, how did you get here? And so we're tracking the quality of the decision making as well. And we can measure, you know, hey, if somebody's overconfident or underconfident, what would have happened to specific investments if you actually had that, if you're actually appropriately calibrated. So that's, that's one way that we do it to make sure that when, you know, first of all, there's an objective score, but also that people are

right for the right reasons. And that level, I think, of introspection is really important. And that's the main way that we track it. So this is a fascinating thought experiment in of itself. but you've actually implemented at Arizona PS PRS. What are the second order effects of this? First of all, intellectual humility just becomes absolutely contagious. So when everyone's score is on a wall, we're all competitive people, right? Nobody walks into a meeting claiming some kind of, you know, certainty that they don't have. So that social vindication element really is looming large.

Second, I think we make fewer big mistakes because the process forces you to articulate, you know, what has to be true for the investment to work. And step one starts with setting what we call the base rate, meaning you take this outside view and you ask what usually happens, what usually happens to say a private equity fund or whatever it is that you're looking at. And that keeps you from falling prey to sort of like, you know, this illusion of control. The third one is, and maybe the most important is that it changes the culture of disagreement. So in some, I think, investment committees, it's like this really combative sort of, you know, gladatorial event where there's a clear winner and a clear loser, or where you have people

who are just really good at debate, who can take up a lot of air in the room, or maybe it's just the senior people in the room talking and the junior people don't, you know, don't speak up. All of that goes away when you have this objective scoring and this, and this objective, very quantitative approach. So it sounds more like, hey, I'm sort of thinking 60% confidence in this investment, but you're kind of at 80. Like, how do we bridge, how do we bridge the gap? Expert calls have always been one of the most powerful ways to build conviction, but today, investors are asked to cover more companies, move faster, and do it with leaner teams. With Alpha Sense AI-led expert calls, their Tegis call service team sources experts based on your

research criteria and lets the AI interviewer get to work. The magic is in the AI interviewer, purpose-built and knowledgeable-based information to conduct high-quality context-stretched conversations on your behalf, acting as a trusted extension of your team. Then they take it one step further. Your call transcripts flow natively into your Alpha Sense experience and become queryable, searchable, and comparable. So your primary insights plug directly into earnings prep, digital work streams, and pitchbooks with zero tool switching.

And with Alpha Sense expert call services, the AI-led expert calls are just one option, because we know the importance of a hybrid expert research approach. AI for coverage and efficiency. Humans for complexity and conviction. It's the institutional edge that scales research without scale. HALING headcount. For hedge funds, that means validating thesis assumptions across dozens of experts before earnings instead of a handful. For private equity, it means faster pre-IOI scans and deeper commercial diligence. For investment banks and asset managers, it means pulling real operator

perspective straight into models and sector positioning without disconnected tools or manual handoffs. All of it lives inside the LFSA's platform, trusted by 75% of the world's top hedge funds alongside filings, broker research, news, and more than 240,000 expert call transcripts, turning raw conversations into comparable, auditable insight. Take advantage of Alpha Sense AI-led expert calls now. The first to see wins. The rest follow. Learn more at Alpha-sense.com slash how I invest. So everybody wants to improve their score, so they're actively looking for feedback from the rest of the team in order to better calibrate their position versus this being the zero sum. This is my investment in somebody being against investment.

And you've kind of butt heads in this year some way. Exactly. And it's very, it's really fascinating. Some of the best forecasters on the team are people who have come to us with absolutely no prior investment knowledge. Now, they're sort of probabilistic thinkers and or they're what we call numerate, meaning they just sort of think in terms of sort of quantitative. But it's not like they're running real complex model sometimes.

It's just that that's, you know, they're thinking in terms of likely outcomes, likely ranges, you know, and again, they start with the base rate. what normally happens and then they start looking for data they can kind of move them off of it and it's not so robotic you know but you can sort of say hey look on average there's kind of a 30 percent chance of these things but i'm looking at this other data and that kind of is moving me you know i moved up kind of five or 10 percent right but then this kind of moving back down five percent so you're you're putting guardrails around a conversation and it's not it's not a subjective and i find that that's just really important i've been going down this rabbit hole

of figuring out how the most innovative people in the world become the most innovative and And prime example, of course, is Elon Musk. I've interviewed a lot of his friends and a lot of people around him. And one of the things that I've learned about how to tell whether somebody's crazy or extremely innovative, one way is, does it actually work? So you could tell a post facto in the beginning, Elon Musk was trying to create a private space company. Even his closest friends, his most innovative friends had an intervention, showed him all these

videos of space rockets crashing. But he really had first principles. So he had these ideas, these building blocks of logic. And he had this thesis on why I would work. And the thin line between whether somebody's crazy or extremely innovative is actually not based on them or even people's reactions. It's based on the first principle's logic. So I know A, I know that sounds crazy. Okay, fine.

Now, B, now look at my logic, poke holes in it, either to improve it and to improve the thesis or to kill the thesis. But if the first principles hold, even though it's completely crazy, then it's actually innovative, not crazy. In the context of our organization, I think that's what this forecasting application really does, because what you're trying to build over time is calibration, right? Are people, when they say they're 90% confident, are they 90% right? If they say they're 60% confident, are they 60% right? So we write down, of course, all the training questions and also all the investment recommendations. And we're looking at those resolutions, right?

And we, for any individual on the team, we can say, well, here are all your 60% confident forecast, right? Here are all your 80% confident forecast. and where you write eight out of 10 on those 80%, six out of 10 on the 60. So when you get to innovation, that becomes really important because what you want as a CIO is to know that your team is calibrated. Somebody comes to you and they say, well, here's an idea we probably haven't thought of. We start thinking through, you know, well, okay, well, is this more like a, is this a 50-50 bet? Right.

And we can get into, you know, how that works with a specific investment example. But like that is really important to help, like we said, help parse the ideas that are truly crazy from the ones that are actually innovative. This reminds me of Bridgewater and how Ray Dalio created the principles where everybody kind of openly fights their positions in order to get better. But upstream of that is if you have this organization where people know that they could come to your organization to basically tune and calibrate their investment skills, that's a hell of a recruiting advantage over other organizations that might be seen as tribal or political.

Now that we do it this way, I can't imagine doing it any other way. Because when you have that level of objectivity, it just levels the playing field and people who wouldn't normally speak up, you know, because they're newer or for any other reason, actually have a voice. And it's not necessarily that they speak up more, but it's the people who can be overconfident, right, that will start to defer. And I think that's just a really healthy team dynamic, for sure. There's this paradox of knowledge where the deeper you go into topic, the more you realize how much you don't know. So set another way, if you don't know a lot, you're going to have a lot of times a very high confidence. So there is this negative correlation between somebody's confidence and sometimes their depth to their knowledge. That's well said because there is like the paradox in decision making, which is the faster you admit what you don't know, actually the better you are at making decisions. And we see this pattern in the forecasting results of our own team. So what will happen is they'll make a few forecasts early on.

and let's say the results are really good. So then they build up all this confidence, right, without really getting any better at the calibration process. And then sort of like mid, you know, mid sort of forecasting career, right? Then they get a bunch of really bad scores. And then they figure out, okay, wait, wait, you know, maybe I was overconfident. So then they start to pull back and they think, okay, on the first ones, maybe I got lucky here.

And I confused that luck with skill. And then I got overconfident. Now you start to break them down. And that's the process of calibration. and they start to pull the confidence down and their accuracy will start to go up. And that's what you want to see. Do you go through this super forecasting process as well?

Let me give you an example of how this would work on the team, right? So you have high, you know, kind of have high and low confidence. And let's just take like a low confidence example, right? So the low confidence example, oh, I always get asked this, right? Can you make a low confidence investment? And low confidence doesn't mean, you know, don't invest. It's just like poker, right? You don't fold a hand just because you're not holding the aces.

So this is classic the way we approach this. Classic expected value theory, right? You multiply the probability of being right times the upside alpha, subtracted by the probability of being wrong times the downside, right? And if the number is positive, you sort of have a case. So like here's an example. I'll make the math easy on myself on the podcast here. So say you're like looking at a private equity fund and it's focused on a specific sector.

We'll benchmark them, say on like a PME basis, again, the sectors like public index. And let's just say for the sake of argument, right, that history tells you in any vintage, say two out of 10, you know, sector specialists will add a thousand basis points of alpha over the sector benchmark. So your starting base rate is 20%. You can be 20% confident with, you know, that's the outside view. What normally happens, 20% confident that a manager is going to add a thousand, a thousand basis points of alpha. And then you go hunting for information that moves a needle. So maybe their prior fund did it. And that maybe moves you up from a 20% to a 30%. Maybe the team's intact. Maybe there's a proprietary sourcing angle. They've got a playbook.

But let's just say for sake of argument, that kind of moves you to 50%, like a coin flip. So 50-50, right? Meanwhile, let's say if you're wrong, you estimate that you lose about 100 basis points relative to that sector index. So you take the average of the other eight managers. And let's just say there are minus, you know, there are minus 100. And so the math is you have 50% times 1,000 basis points, so that's 500, minus 50% times 100. So your expected value is 450 basis points. That's not certainty, but that's a solid investment. And that's basically how we approach it. Last time we chatted, you mentioned that you're being benchmarked against the S&P 500, whether you like it or not. Why is that? So here's the uncomfortable truth of institutional investing.

Every dollar we put into private equity, real estate, hedge funds, you know, what have you. It's a dollar. We didn't put the S&P 500 index or a bond index, but nobody compares alternative investments to the bond index. So the S&P wins the benchmark conversation like just by default, probably because it's like the one number everybody already knows. It's familiar. It's on the news every night. So even if it's not your explicit stated benchmark, it's the one in the back of every journalist's mind, every legislator's mind, every board member's mind, every constituent's mind. And like the S&P has been, you know, extremely good for the last 15 years, which makes the conversation even harder. And my job is to justify the complexity, the illiquidity, the fees of a diversified portfolio.

And the only honest defense is long-term risk-adjusted returns over full market cycles. And we're not trying to beat the S&P every year. We're trying to not be the fun that blows up when the cycle turns. But like, as I say all the time, diversification is a absolute punishment for sure in a bull market. but it's a miracle in a crisis. And it gets even more complex, given that you manage for police and firemen, so it's not just one single decision maker

where you could basically explain and educate them on the market. There's thousands of underlying members that are de facto going to use SB 500 as this mental benchmark. Absolutely. And you have to keep your audience in mind. And that you just,

and I think it's good that you sort of keep, you know, keep that idea in the back of your head because it does keep you really honest when you're adding investments. And I think that's not a bad way to construct a portfolio. I do think you have to have to have kind of like what your constituents understand in the back of your mind at all time. So as annoying as it is, I think it is a healthy way to build a portfolio. And I think it's a fair way from a public policy standpoint to interact with your constituents. And you have to construct your portfolio in these two co-centric circles, who your audience is and what is the ideal portfolio and find a way to kind of match both of those?

by and large, you want to kind of keep the portfolio constructed in a way that people will understand it, right? I mean, that's the, I think that's really critical. I mean, these are people, right, who didn't sign up to steady asset allocation and their retirement is not in their own hands. And it can be kind of disconcerting to kind of like trust other people and, you know, with your retirement money. So I do think you have an obligation to construct something that they understand. Now, at times, there are things where you just know, look, this is not going to be well understood, you know, to a lay person, but we feel strongly that it's going to be helpful in the portfolio. and then you know you had that conversation with your board but I do think you have to kind of keep what other you know what your constituents will understand in the back of your mind at all time

tell me about your portfolio construction today and maybe how that's evolved over the last few years so when I became the CIA in 2019 we had something like 10 like we 10 asset classes I think is what we had maybe more so and I've changed that to simplify it remarkably so we start with a really simple premise right our liability at the end of the day there is a long standing stream of payments to first responders, right? And so for me, the mission is in this order, one, have money when money is due, managed portfolio volatility. So it doesn't translate into contribution rate volatility that employers and members just, you know, can't absorb because it makes it hard. And do all of that for, do all of that, I think, spending as little money as possible. That's

the whole job in that sequence. It's like I said, we do feel an obligation to make, to make it understandable to the people whose retirement this is. Again, they're not signing up to study stocks and bonds. They signed up to serve. So we really break the portfolio at a really high level into three broad categories. And for us, that's called capital appreciation, contractual income and diversifying strategies. Because those labels, I think, describe how investments actually behave, not sectors or anything like that. And we tell members, within each of those, there's publicly traded investments and there's private ones. That's basically it. So the edge in my mind, isn't having like all these exotic buckets. It's just having fewer of them and filling the buckets

really well. You can think of the buckets as essentially constraints. So if you have a specific public equity and private equity and for some reason you see private equity as more risk adjusted, then you're arbitrarily constrained into this public equity where you don't want to be playing just because somebody came up with these buckets. I get asked why I have the loosely defined buckets. And I think to your point, I think the rigid buckets can, I mean, the glib answer is the rigid buckets can lead to just like stupid decisions, right? So I think the problem with the finding asset class is by like sectors, having an asset allocation, I mean, where you have sectors and sort of capital structure. So take, you know, you have a real estate allocation, but then you

also have a public equity allocation, right? So you get into these conversations about, what do you do within a real estate equity guy or or a real estate credit guy, right? And I just think that that gets conceptually messy pretty fast. So sure, real estate, you know, sometimes behaves differently than stocks, but how often does it really, right? And with like what degree of confidence? And take probably the most recent example would be commercial real estate, right? Commercial real estate has been like a long time darling of the inflation protection crowd. And then COVID hit, followed by like actual inflation. And commercial real estate didn't help because other forces were swamping the inflation signal, right? So the theoretical benefit just got buried in the specific like facts on the ground.

I think making the buckets wider, it increases competition among investors. and ideas, and for a pension fund with the longer horizon, where we can stomach some of that volatility, the more competition for capital, I think just leads to better outcomes fundamentally. It's a loosely defined ranges, lets us be flexible and opportunistic, which is a huge advantage. The buckets are, in our mind, it's just kind of a map, but like, you know, any navigator knows the map's not the territory. Dell PCs with Intel inside are built for the moments that matter, for the moments you plan and the ones you don't.

built for the busy days that turn into all-night study sessions, the moment you're working from a cafe and realize every outlet's taken. The times you're deep in your flow and the absolute last thing you need is an auto-update throwing off your momentum. That's why Dell builds tech that adapts to the way you actually work, built with long-lasting batteries so you're not scrambling for the closest outlet, and built an intelligence that makes updates around your schedule, not in the middle of it. They don't build tech for tech's sake.

They build it for you. find technology built for the way you work at dell.com slash dell PCs built for you Dell PCs with Intel inside are built for the moments that matter for the moments you plan and the ones you don't built for the busy days that turn into all night study sessions the moment you're working from a cafe and realize every outlet's taken the times you're deep in your flow and the absolute last thing you need is an auto update throwing off your momentum. That's why Dell builds tech that adapts to the way you actually work, built with long-lasting batteries so you're not scrambling for the closest outlet, and built an intelligence

that makes updates around your schedule, not in the middle of it. They don't build tech for tech's sake. They build it for you. Find technology built for the way you work at Dell.com slash Dell PCs. Built for you. If you've been considering futures trading, now might be the time to take a closer look. The futures markets has seen increased activity recently and plus 500 futures offers a straightforward entry point. The platform provides access to major instruments, including the S&P 500, NASDAQ, Bitcoin, natural gas, and other key markets across equity indices, energy, metals, Forex, and crypto. Their interface is designed for accessibility. You can monitor and execute trades from your phone with a $100 minimum deposit. Once your account is open, potential trades can be

executed in two clicks. For those who prefer to practice first, Plus 500 offers an unlimited demo account with full charting and analytical tools. No risk involved while you familiarize yourself with the platform. The company has been operating in a trading space for over 20 years. Download the Plus 500 app. Trading and futures involves the risk of loss is not suitable for everyone. Not all applicants will qualify. Are you essentially implementing TPA into your portfolio and if not, why not? I'm going to be honest. I can't describe TPA.

And every time I hear somebody like define it, I just think like that kind of sounds like what we're already doing. And, you know, maybe we all arrived at it independently. And maybe that's a sign, right? Maybe the logic of it is pretty sound. But I think either way,

the core question is, you know, within each sort of bucket or sort of risk category, what's the best complement of assets available to us, you know, right now, given our side, it's not like,

hey, is this a, you know, a good credit investment or to your point? Like, is this a good private equity deal? and maybe, you know, private equity is just not all that attractive? But it's like, is this a better use of capital than the next best alternative with a similar risk and correlation profile? And so I think that reframing is just really, I think really important. And I think it's generally hard to execute institutionally because it just requires overcoming this gravitational pull towards asset class silos, which has just run, you know, a huge deep trail down all of our cerebral cortexes. People get comfortable

defending their buckets, right? And I think, like my understanding of TP, is like you're just, you know, you're trying to compete across all the buckets all the time, right? And pair things with common risk exposures to things, other things with common risk exposures. One of the benefits of this bucket at approach or endowment approach is that you have your best athletes in specific asset classes. So you have your real estate credit, your real estate equity, you have your private equity, public equity. How do you build a team that's able to be so flexible? And how do you build incentives mechanism in order for your team to be flexible? That's a great question. So I think the answer at the end of the day is,

You know, you just have to hire the right people. Public funds have, like, various constraints. You have to hire people that are intellectually honest. You have to hire people that just never stop asking questions. And the nice thing about having fewer buckets is you don't have anyone that's territorial, right? I can say to someone on my team, hey, listen, you've got to manage the private equity investments right now. But let's be clear, this could be 0%. There's nothing obligating us to have any money in this asset class.

right so i think the good thing about that and in fact i love the the pm that we have who's running real estate right now he has spent probably the better part of three years just saying no to everything but that hasn't been his only assignment he's worked on a bunch of different asset classes so i think you just have to you know going in tell the people look you know you're going to be athletes you're not going to be married to a single asset class and that just that just makes people really honest and it's not hard for people to say look i don't think there's anything to do here in fixed income right now. So I'm not, I'm not making any recommendations or real estate where we've just been saying, no, not re-uping with any of our partners. And, you know, nobody's worried about their

job. It's all about what's the best allocation, you know, if we're going to take this kind of risk, where, where's the best opportunity in that market right now? And I think that's just a better way to run the portfolio. And it's, I think it's resulted in improved performance for us, for sure, when I started as the CIO, we were in the bottom desial pretty much across all the time horizons. and now we're in the top third across all the time horizons. And I think you're having the right people is part of that and then also simplifying the asset allocations. So you have greater competition for deals because it just raises the bar.

How big of your strategy is co-investments and has that changed? So we haven't been real active co-investors in a lot of the credit like instruments within what we call global private equity, which encompasses private real estate, private real assets, buyouts, venture capital, all of that. stuff probably about 10 to 15% of that allocation. So it's probably maybe 5% of our overall portfolio and increasing. We have a big focus on just trying to get closer to the assets. Co-investment is going to become the more important part of our program for sure. When I interviewed Chris Ailman, who is CIO of Calsters for 23 years, he said one of the ways

that they really generated alpha was structural alpha in co-investments. And one of the only things that really worked in co-investments was rules-based co-investment. So creating a diversified pool of co-investments, lo and behold, the alpha there was the fee savings. So it's somewhat of an interesting view on private equity in general, but they were able to use their size to an advantage and capture this co-investment structural alpha. It makes a lot of sense. If you focus on a handful of partners, we tend to like the sector specialist. And co-investments, I think, from sector specialists are just of a higher quality than say maybe a more generalist or diversified manager. Now, like obviously, there's exceptions here and there. But I think that's not a bad way to do it. And what you're

effectively doing is just creating your own private equity fund, but the deals have already been vetted, you know, by a GP that you've underwritten and trust. And so you can have effectively a private equity program with substantially discounted fees or no fee and no carry. Even if they perform in line with all the other deals in a private equity fund, you've maybe increased your return on those by three or four hundred basis points because you've eliminated that gross, gross to net spread. You're one of the best first principal's thinkers that I've interviewed in terms of CIO. I want to get your opinion on something. So I've been thinking about this concept of the semantics on, call it investing. Some people call it allocation. Other people call it investing. And then I had

Sam Zell's long-term partner, Mark Soder, and he calls it own, owning. He said, Sam Zell never said, I allocated or I invested. He said, I own this asset. Something about that to me feels more powerful when you're thinking about which assets do I want to own, not where do I want to allocate a way, do I want to invest? Do you think there's something there? I do. I do. I do. I do. I do. I do. I do. I do So in fair disclosure, I have not heard that, but I've always hated the, I've always, always hated the term allocator, right? I mean, I don't see much of a difference, whether, you know, you're buying a stock or investing in a fund. P.SPRS, we do everything. We've got some, we've got some direct investments. We can buy and sell stocks and bonds. We've got ETFs that we buy

and sell. We obviously commit to private equity and private credit funds. We've done co-investments. But even if we didn't have all of that, and we were just allocating to funds, the practice is the same. I mean, it is an investment for sure. And I think we, I think allocators becomes just more of a semantic. I mean, we're just trying to distinguish between the players in the market more than anything else. But I like ownership. I like that idea. I'm going to have to use that and I'll give full attribution to them because obviously for me, accountability is a huge thing. To be fair, this podcast is how I invest. If I could call it how I own, if that sounded better, we would also call it how I own. That's true. But I have to be intellectually honest, I do think

own is stronger than invest. I think invest is still passive. You could say like maybe synantically like ownership is like maybe maybe something was given to you or you know, you inherited it or something like that. So it wasn't maybe necessarily a certain, you know, a decision with an outcome that you have to realize. But yeah, I mean, I certainly prefer invest over allocate any day by like, I like the ownership idea just because again, start to finish, you own the work and you own the outcome. And I think that that's your semantics, you know, words matter. And it's important for enforcing behavior. So I do like that idea. What's something you've changed your mind on in the last 12 to 24 months?

I'm anchoring to the most recent geopolitical events in some of the conflict in the Middle East. But we've been on a rampage to really humble the confidence of our investment partners. I probably would have thought three or four years ago that the confidence that our partners have in their specific outcomes were merited. I no longer think that as we're looking at our managers across the board because we do sort of implicitly benchmark them using sort of the same kind of forecasting process that that we have internally. And a lot of them are overconfident. And I think, you know, that's been a fundamental shift in my mentality is that, you know, we, it's not, it's not a 50-50 whether they can do

what they say they do, but like we have to be a lot. I mean, the default has to be, look, there's a, you know, maybe a 20 or 30% chance that they can do what they say they're going to do. There's a lot of marketing, but I think treating partners with a much higher level of scrutiny in this environment is really important. So I've gone from thinking that, you know, look, they probably have a pretty good handle on things to just thinking, certainly in some conversations that, you know, there's marketing. I understand that from a business building standpoint, right, that certainty is what sells, you know, or if you're on the other side of it, pessimism, if you're shorting things, pessimism just always sounds smart. But I think what everybody has to do

right now is just reduce their certainty and whatever they're doing, whether they're long or they're short. And I think that's critical. there's such a performative aspect to this confidence we just invested in a company called lagora and lp was asking me about horizontal versus vertical AI i have this itch to say really nobody knows how could you know horizontal versus vertical AI and if mark andrescent is on the record saying he doesn't know peter teal doesn't know why would i know but you have this pressure to justify this position and everybody expects it you have to basically justify this unknowable variable without saying the more intellectually sound thing, which is, well, if vertical AI wins, or in the 50%

of cases that wins, this has an expected value of 5x. But in half the cases, this might be a 1x or it could even lose one. I would be in that camp. I think, I would think, look, I think vertical AI is probably like the way to go. It's probably still where I think things are going because, look, with vertical AI, it's a huge moat, right? It takes, it takes a lot of resources to just understand the sector because every sector is extremely complicated. So there can be a strong case made for that. But I think with AI generally right now, right, everyone has to be careful.

We don't know the full impact of AI. And I would be real hesitant to take a view on whether horizontal or vertical is going to be the winner. And I love the way that you position it, right? I mean, if these guys were the experts in the field don't know, then I think it's fair for us to say we don't know. And there's a similar discussion going on with like home builders and interest rates. You know, if you listen to like the earnings calls of home builders, you know, they don't really have a clue where the housing market and interest rates are going. They can kind of speak for like, you know, the next year or so.

But if they don't know and you would think that they're probably as close as anyone to it, then you have to say we don't know. And that's really important in terms of how we make decisions when we're trying to create those base rates. I'd be real hesitant if anyone on my team said, hey, look, I'm 60 or 70% confident. I know the interest rates will go up by at least 100 basis points in the next year. Or I'm 60 or 70% confident that horizontal AI is going to win because you'd have to convince me that like the base rate right now, according to the experts, is 50-50. It's a coin flip and that's max uncertainty. And you'd have to point to some pretty substantial evidence that would move you off of that. We all have to be very humble right now about what we know and what we don't know.

And like I said at the beginning, the paradox of improving is just admitting, what you don't know and the more you do that, then, you know, I think the better, the better you are about making decisions. Now, that doesn't mean you're paralyzed, right? But you have to approach investments and say, look, I'm 50% confident. So if I'm looking at this investment on an expected value basis, meaning I'm 50% right, or do I think I gain if I'm right, you know, times like the negative alpha if I'm wrong and, you know, 50%. So you could still be making investments into that, but I think you have to say this is a coin flip, as close to a coin flip. And there's another sort of behavioral component to this too, which is we're often really good cognitively,

sort of the yes scenario, will horizontal AI win or will vertical AI win? Because you can kind of point to various things and you can build a case in your mind because our human brains are reasonably pretty good at that. But what we're not good is generally the no case, right? So will horizontal AI fail to win or will vertical AI fail to win? Because that's really cognitively expensive. You have to like look at all the reasons it possibly could and just blow them all up. And so it is a delicate balance right now. But I would certainly be closer to that 50-50 threshold. Just a pressure tells us that, do you have GPs that come and pitch you in probabilistic manner? And if so, how do you react to that?

No. I would say not usually. You know, the conversation with GPs, you know, usually goes something like this. We aim to perform over a market cycle, right? So then our team's job is to pin him down and say, well, what's like a market cycle for you? Well, like five to 10 years. Well, which is it? Five or 10, right?

And generally partners are, you want to keep that. flexibility because right you know we all understand you the the future's uncertain and you know there's compliance and things like and things like that but it's also a huge problem in the LP community if you can get your potential partner to define like what success looks like then any result is going to do right this is the classic just drying the bull's eye around wherever the arrow landed so I think what you have to do is say hey listen I need a minimum a minimum success definition here like what you know say well we're looking for mid teens over market market cycle okay well like what is that like 15 13 13 18 18 18

18. So you pin him down on that. Well, okay, at least 13. All right. What's that market cycle look like? Well, it's not really a market cycle. We're trying to beat the benchmark. Oh, okay. So it's a relative performance. So our team, their job is to just sit there and pin down. What does success look like? Because I maintain as an LP, you're never going to be the expert in any one area, right? You're talking to venture capitalists. You're talking to CTAs, right? You're talking to CTAs, right? You're talking to global macro funds. And I maintain, if they can't define success, that's problem number one, right? Two, if they're defining success and you finally get them to say, because we do have GPs that'll do this. And that's sort of like the threshold for getting into our portfolio. They'll say, look, we're looking for a 12 to 15% return on a rolling 18 month basis, about 60% of the time.

And then we're just watching them. Every 18 months, we roll it forward. Hey, are you hitting that return 60% of the time? And I maintain, like, since because you're not an expert, you don't have to know everything that they're doing. but if they're miscalibrated and they say 60% of the time, we're going to hit that return threshold, but they're only hitting at 40% of the time. You don't have to know what they're doing. You just know that they don't know what they're doing because they wouldn't be that far off.

And in fact, you see a lot of GPs who are overconfident. And that's a problem because if you go back to the expected value construction, if you think your success is 60% and it's actually 40, well, then your equation has a negative expected value and you end up with performance disappointment. You're essentially forcing super forecasting on your GPs. Yep. Yep, well, we will impute it and we will ask them directly. You know, so if it's a liquid fund, we'll certainly ask them that if it's a private equity fund,

obviously we're getting sort of like a whole, we kind of walk through these sample I gave earlier, right? But then you can also go portfolio company by portfolio company. And you can say, hey, this portfolio company, where do you think revenue is going to end up 12 months from now? And they'll say, kind of here. And you say, well, how confident aren't you? Ah, we're 60%. Right. Okay.

Portfolio company B, what do you think? Right. And then fast forward a year, you know, you can start to compile some data on. them and you can say, look, you guys are actually pretty well calibrated or you seem to be routinely overconfident. And I just think LPs really want to work with groups who are appropriately because then they're sizing bets appropriately. And that's a little bit how we do it. If you could go back to 2007, it just left J.P. Morgan and joined PSPRS. What is one piece of timeless advice you'd give a younger mark that would have either accelerated your career or helped you avoid

constant mistakes? The number one thing is to communicate to any young investor that most of what you look at will probably not do what you think. So when we first started doing, when I stepped into the CIO and we were forcing everyone on the team to assign, you know, sort of confidence levels to investments, the confidence levels were really high, 75% confident that this fund is going to achieve the success definition. It's 60, 80%. And it was a bit of an awakening. Let's say you're looking for top quartile private equity funds.

Okay. Well, then that means by definition, most of the people across the table from you are not going to be there because only 25% of them can be and 75% of them will not be. So what I hear from other investment teams or individuals, they sort of kind of go into these meetings thinking like, okay, this sounds like a good idea. I'm going to listen to them and then I'm going to try to poke holes in what they're saying. But that's coming from a position like you're giving them, you know, you're basically saying they're going to do what they're going to do and I have to find information that disproves it. or they'll come in and say maybe, which is a little bit better, like, I'm going to be on the fence. I'm kind of 50, 50. Let's see if they can convince me.

But it's completely different to sit there and say, there's a 25% chance they're a top quartile fund. Most likely they will not be. And they're really going to have to convince me. And I've got to point to very, very specific criteria, right, that would move me from 25 to, say, you know, 50 or 60% confidence. And I just think that's a healthy frame. But I think it takes some time. But doing that and again, starting with that out.

outside view that most funds actually do not perform, I think is a much healthier way to approach investments. And when I was back in 2007, when I was starting, I didn't realize and understand that. I do now. And that's one thing I would definitely, you know, beat into my 2007 self from the get go. So that's slightly a different way. The market, when people look at historic returns, they look at NASA class and maybe returns 8% or 11%. They intuitively think it's just like this linear growth. But especially in the most bikey assets like venture capital. Some years you're getting 40%, some years you're losing 18%. It's really quite chaotic back and forward. So what do you do with that? You do two things.

One is you have to discount any kind of projections. So if you're trying to get top quartile, you probably need to pick a top death off fund. If you're going to pick a top quartile, the odds of it staying top quartile, according to the University of Chicago, Steve Kaplan, is 52%. But that also says 48% of the time, you're not going to get that. So you have to be more ambitious in order to hit your target. And then secondly, is when you're right, you better make a count, you better make sure that base symmetry is there because if you have just an asset class that is going all over the place and the median and the mean is basically the same amount, then it's not a good asset class.

So part of this is just sort of when we start with this outside view and creating base rates, what you're really trying to do is kind of control for skill and luck, right? And so that's why it's really important to pin a potential partner down and ask them to define the success metrics. I'll use this analogy all the time, like, you know, take the University of Michigan Stadium, right, biggest stadium in college football. If you had everybody stand up in that stadium, and if you flip heads, you stay standing and you flip tails, you, you sit down.

You do that 15, 15 times, 20 times. I don't know the math off the top of my head, but it would be 0.5 times 0.25 time 0.5. You'd probably end up with a handful of people standing at the end of maybe, you know, 15 coin flips. But it's not like they did that intentionally, right? But you can imagine like what happens. Like imagine they were investment, you know, they were like venture capitalists or whatever. You know, they had some home runs. But what you don't know is whether that happened on access.

or whether that was skill. Going back to my analogy, imagine that they say there's five people. Imagine that those five people or one of them said, I'm going to flip 15 heads in a row ahead of time, right, before it started. And they actually did it, right? That would matter a lot more. So I think defining that success in the outcome beforehand really helps you interpret the results because otherwise there's a lot of revisionist history, right? And they'll take credit for anything that turned out really well, right? And they'll sort of, they'll sort of just like, you know, uncritically accept that. And they'll assign bad luck to anything that didn't turn out well. So I think from the jump, having that definition of success is just really important.

Well, Mark, it's been two years since I've been trying to get you on the podcast, ever since you won Innovator of the Year for Institutional Investor. Congratulations on belated on that. And thanks so much for taking a time and for sitting down. Hey, no problem. This has been fun. Thanks for having me. If you found this conversation valuable, please click follow how I invest so that you don't miss the next episode with the world's top investors.