Invest Like the Best with Patrick O'Shaughnessy - Jeremiah Lowin – Machine Intelligence and Risk Management - [Invest Like the Best, EP.20]

Episode Date: January 17, 2017

Jeremiah Lowin is probably the smartest guy I know, and that is saying something. He is an expert in the fields of statistics, artificial intelligence, and risk management—among many other things.  He is currently the Director of Risk Management for a private investment firm in the New York area, but has spent years working with machine learning and AI.  This conversation is broken up into two parts.  In the first part, we explore artificial intelligence, machine learning, and models.  Then we shift to what risk means in a portfolio and how it can be managed or at least redistributed (which starts around 40 minutes into the conversation).  Please enjoy!   For comprehensive show notes on this episode go to investorfieldguide.com/lowin/ For more episodes go to InvestorFieldGuide.com/podcast.  Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub Follow Patrick on twitter at @patrick_oshag

This podcast is sponsored by CFA Institute, the Global Association of Investment Professionals whose mission is to lead the investment profession by promoting the highest standards of ethics, education, and professional excellence for the ultimate benefit of society. CFA Institute serves a global community of investment professionals working to build an investment industry where investors' interests come first, financial markets function at their best, and economies grow. The chartered financial analyst credential is the most respected and recognized investment management designation in the world.

The views expressed in this podcast do not necessarily represent the views of CFA Institute. Hello and welcome, everyone. I'm Patrick O'Shaughnessy and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, methods, stories, and of strategies that will help you better invest both your time and your money. You can learn more and stay up to date at investorfield guide.com. Patrick O'Shaunisee is a principal and portfolio manager at O'Shaunacy Asset Management. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of Oshamacy Asset Management.

This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of O'Shawnessi Asset Management may maintain positions in the securities discussed in this podcast. My guest this week is Jeremiah Lowen. Jeremiah is a childhood friend of mine who has been a sounding board for me throughout my career. We have conversations like the one you're about to hear about once a month and in all those conversations just like this one, you can hear me just trying to keep up. Jeremiah is one of those perfect SAT score guys, literally, who talks about topics like artificial intelligence like he's placing a lunch order.

His career has been in risk management, and he is currently the head of risk management at a private family office in the New York area. The conversation is in two halves. In the first, we discuss models, machine learning, and artificial intelligence. In the second, we talk about what risk means in a portfolio and how it can be managed or at least redistributed. You'll have to listen pretty closely to this one, but if these topics interest you,

it's a chance to see one of the leading minds in the fields of data science, machine learning, and risk management at work. Please enjoy our conversation. So Jeremiah, starting 16 years ago, you were teaching me how to do geometry proofs, and I've been learning from you ever since. And so I figured, like all the lunches we have over the years,

that this podcast is basically an excuse to have conversations like we tend to have in a more public forum. So thanks for joining me. Thanks for having me. The place that we'll start and the two major themes that we'll talk about today are learning, very broadly speaking, more specifically in the areas of machine learning and artificial intelligence, but also risk management in the investment process. So we'll start with learning.

And I always have to remind myself when thinking about these more complicated topics that there is no conscious machine intelligence yet, at least not that we know, of. And so there are still very bright people behind the systems that are doing all of these things. And so it would be fun to hear kind of your own story. And we'll use building blocks, just like a child learns through orienting themselves in the world and tinkering around. We'll try to use some building blocks to get up from more simple to more complicated topics. So maybe just give me a quick bit of background about how you got started in, I guess, the foundational elements that allow you to explore things like artificial intelligence?

Sure. So for me, that actually goes back to stats 100. We were given the sort of canonical example of a normal distribution being the stock market, stock returns. And I happened to be reading the misbehavior of markets at the time and of course read that the stock market is anything but normally distributed. And it was the first time that I recall that I had this very academic dissonance where one, you know, pseudo professor through a book was telling me one thing, and my real-life professor was telling me something very different. And the light bulb went off, and I said, well, I've been given the tools to actually go out and answer this question. So, of course, I did, and it took some time. And of course I learned what you and I know well, which is that the

stock market is anything but normally distributed. And I went to my professor, and I said, listen, I don't understand. You said one thing, and I've demonstrated another. And she told me, and I'll never forget, that sometimes in life we make necessary approximations. And that was my first encounter with building a model and understanding when you'd built a good enough model as opposed to going too far.

And for Stats 100, normal distribution was the right level. And that kicked off for me just this quest for building models, exploring models that was in turn would lead me into machine learning, which is just a more sophisticated version of that. Of course, that led into finance, which is where I've always kept one foot, you know, on one side of that line, one foot on the other side, and I always try to meld the two.

This idea of models is so interesting where a model is effectively an analogy or a metaphor, just like language is. So that's why at the beginning of the Tao Te Ching, it says the Tao that can be named is not the true Tao, because language captures some large percentage of an underlying essence or truth, but it doesn't capture at all. So my favorite example for this, I've used this before, but I'll use it again because I'll use it a little bit differently, is trying to equate concrete and abstract things.

So you're trying to understand the abstract and you do it metaphorically by referring back to a concrete thing. So if you say an acre, it's a very abstract construct or idea. But if you say a football field with the end zones cut off is an acre, people, you know, there's a click of understanding. that's a model. The problem is that there's an error term in there, right? So a football field is actually about 4,500 square feet bigger than an acre. So you get to 10 acres and you're an acre off. And that's how I think about, that's how I think about models, is imperfect reflections of an underlying truth. So, so what was your next step? Was it from Stats 101? What about finance? Was it finance first? Was it this, I mean, obviously Mandel Broad is a big influence on both of us.

Was it finance or markets first that got you to the next step? You know, it all happened by accident, and it all happened at the same time. I wound up working at a firm called Amaranth. Amaranth famously exploded in 2006 when that was still unusual for a hedge fund. I was interning there, and I was so blown away by some of the people I was working with, advanced degrees in statistics and mathematics and computer science, and they had this common vocabulary that I lacked. And one of the drivers for me was simply,

how do I work with these people? How do I raise myself, my intellect, my understanding to their level? And the answer seemed to be, well, you go out and you get an advanced degree. Now, once I started exploring statistics, which is where I ended up doing my master's, I learned about computational statistics, I learned about empirical modeling, I learned about all these wonderful things that basically boiled down to,

go out, get data, build the best model you can, and then figure out how to use it, maybe not necessarily in that order. So unlike this stock market normal distribution example, I can't tell you there's one thing that sort of propelled me into this, except just this drive of passion for lack of a better word. It fascinated me, it continues to fascinate me. I spend who knows how much of my time, just trying to find better, better ways of building models

without going too far down that rabbit hole and dangerously over-modeling the world. If we think about the simplest model as just like a linear regression, like a simple equation, where you've got a bunch of data points and effectively you're literally drawing a line through it that minimizes the error,

that minimizes the distance between the average distance between each of those dots and the line. How do we start to get from that, which we can all do pretty easily in Excel or whatever, to some of these more complicated buzzwomen wordy ideas that that seem to dominate headlines today. It seems like it's like mundane process, machine learning, profit. And if only it were that easy. Yeah. And so, so help me understand. You've started to, but maybe we'll start a little, we'll go through it again, how machine learning

relates back to the simplest of models and then building up from there. Yeah. So at the risk of grossly oversimplifying the whole thing, but in order to give a flavor of, I think the answer you're looking for, linear regression is a great place to start. We have X's, we have Y's, or rather we have inputs, and we have outputs or desired outputs, and we're seeking to build a model that relates them, that turns X into Y. So linear regression is just such a great utility tool for basically every statistician to keep on the shelf. You reach for it whenever you need it. It's a closed-form solution, which means you don't really have to do much work.

It's well-understood. You line up your X's, you line up your Y's, and boom, the relationship between them falls out. And the reason it's so easy is because it is explicitly defined as a linear relationship, which just happens to be very easy to work with. So we can take steps past that. We can start to introduce nonlinear relationships. We can start to introduce a more complex correlation structures. As many steps we can take, some of which retain this sort of nice, easy, closed form nature, meaning has a solution. We don't have to do much work. And some of which start to get into what are called iterative models. Iterative model,

means we know what we're trying to do, and we have a way of quantifying how good we are at doing it. We usually call that the error, or the error term. And we're going to take steps towards that. Machine learning, as a catch-all term, basically relates to a class of iterative models, meaning we don't really know this closed-form solution for them. But we have well-defined error terms, and we have well-defined methods of overtime decreasing the error. So I line up my X's, I line up my Ys. But now instead of linear regression, I apply the flavor of the day from the machine learning toolkit,

and it'll run through the data, and it'll come up with an answer. And the answer is probably wrong. It's almost certainly wrong, because we haven't built a model yet. But it'll analyze its own result, and it'll say, well, if I tweak this parameter and raise it and I lower this one over here, I get a better answer as measured by whatever this metric we've chosen to quantify our error with. And it'll do that, and it'll do that a million times. I'll do that a billion times. You can overfit the hell out of these things.

Thinking about the actual mechanics of how it works and the fact that there is learning in the title is interesting to me. So a couple of things that maybe are apical. You tell me. One is this idea of brute force that you just try a lot of things, and we can do that a lot faster with computing power now to steadily reduce those error terms. And the second is evolution.

That effectively what evolution does is tinkers in the form of, mutation, it finds things that work, and then it amplifies on those. So is either of those a clean metaphor, and if so, is one better than the other for what machine learning is doing with datasets and its variables? So typically machine learning will actually take a third option, which is a more optimization-driven approach. On the brute force side of things, brute force works, but it tends to waste a lot of time, right? You have a giant search space, your answer is somewhere in it, but you're going to spend an awful lot of time looking at the places your answer isn't. And that's wasted time. That's wasted processing power. Generally,

we don't like that, right. Evolution, there are many, many evolutionarily driven approaches. So you can evolve an answer. You start with, for example, 100 answers, and you see which ones were good, and you take certain characteristics of those potential, we call them candidates, and you throw away, let's say, 90% of them, you keep 10%, and now you breed them. And what you literally do is you say, okay, well, this answer was doing a lot of addition, and this one was doing a lot of multiplication, and the multiplication answer was more effective. So in our next generation, let's do a lot of multiplication.

Again, I'm vastly oversimplifying here. But you literally breed your good answers together to hopefully create yet more good answers. And you can evolve the answer to a machine learning problem or to any problem. What has happened in the machine learning world, rather than that, which has a very, it's still very computationally intensive,

you may end up with answers that are awfully weird just because they evolved in a strange way. The machine learning community has evolved, no pun intended, toward optimization-driven approaches where the derivative of this error term is well known. So if I know, if I have a function that tells me how wrong I am, and I know the derivative of that function with respect to my input parameters,

that's how I know how to tweak my model. And so intuitively, we like, like that approach because we know at every step what it's doing. Given a set of X's, a set of Y's and an error function, I know when I run the model how to improve it. The evolutionary approach doesn't, we don't know how it's going to improve. There's a lot of randomness and how these things breed. And the brute force approach is just going to try everything. So this is sort of a middle ground which is appealing because to some degree it's interpretable where the other two approaches are not. Let's use a concrete example, one because it's an interesting one, but also because

there's a huge piece just this morning in the New York Times about it, which was a great story I'll link to it, about how Google has used some of these machine learning and artificial intelligence concepts to improve their translate function. So maybe, I think we both read it, maybe through that lens, describe a little bit more concretely how these things are being applied to improve a function like that and why it's effective. Sure. So Google, it's hard to think of a company that's done more than Google to effectively communicate the benefits of everything we're talking about to consumers. You use a Google product. You are taking advantage of very concrete implementations of what we're talking about. That said, the examples in the article specifically

translate are using really, really, really, really cutting edge stuff. And the reasons that it works necessarily better than, for example, what they were using a generation ago is still very much out in the tail of the details of this implementation. But if we want to characterize it as a high level, what Google's doing with translate is statistical learning as opposed to rule-based learning. So there, it's a little more like a Rosetta Stone kind of approach, right? So you dig up this Rosetta Stone, it's got two languages on it, or three, I think, historically. And for the first time, you look at them side by side and you're looking for patterns. At the end of the day, machine learning is just pattern-seeking algorithm.

Google's challenge is without telling the system that the word and has such and such a grammatical form and, and this is how prepositions work, and verbs have to end in such and such a way. They want to let the computer discover these patterns on its own by presenting it with a piece of text in one language and a piece of text in the other language where you're telling the system these two pieces of text are the same. They have the same semantic underlying meaning.

The challenge for the computer is to take these characters, which it doesn't know what they are, letter A, letter T, whatever it is, and find some representation that applies to both of them. So that when you say to the computer, I'm giving you some, string of characters in English, it builds this representation, and then it can use that representation to generate the same in French. I realize I haven't actually explained how it does that, and I'm

just trying to think of an easy way of getting into it. Let's say the English is the X, and the French is the Y. And we're asking the computer to map from one to the other. Now, it has to look over an enormous number of such pairs to be able to do that because of homonyms and synonyms and all kinds of weird linguistic hurdles, yeah, that it has to overcome. But in the aggregate, as it looks at English text and French text, it will start to find statistically things that seem to go together or patterns that seem to go together.

That's the approach that Google started, I don't know, 10, 15 years ago with their first version of Translate. What the article references that has led to this incredible leap in accuracy is the layering of time into these models, which is a particular interest of mine. So you use what's called a recurrent neural network. A recurrent neural network essentially means that rather than just taking a batch of X's

and a batch of Y's, where the relationship of the X's to the Y's is all we care about, we're going to start working with ordered data. So for folks in finance, this is critical stuff. As soon as the order matters, everything changes. So if I give you a book, but all the pages are out of order,

it probably means nothing to you, even though it's the same text or a movie with all the frames scrambled, it means nothing to you, even though it's the exact same information as if you sat down and watched a movie. So the insight here at a very high level is that the order is really important because order equates to context. And so as Google goes from just looking at a series of words and trying to relate them to another series of words, to actually saying, no, no, no, let's step through these words and build up context for this sense. Just as you and I, as we're talking, are building up a context for this conversation, for this question that you've asked me, for the last word I just said. Google systems are doing that, and in doing so build up a much more powerful representation of the

information they're trying to model than if they just try in one shot to take this series of words and just figure out what it means. So you built something like this. It was called the Loan Data Company, and it was dedicated to kind of a business. applying these techniques to time series. So tell me a little bit more about that because when you go to the site, which is still live and I think still important, it says we make time machines, we make intelligent machines.

And this opens up a huge and important question, which is if we are able to replicate and actually improve upon the way that we learn. And my understanding is that there's a lot of different techniques for actually doing this. But effectively what we're doing is replicating what has worked for us. You said pattern recognition before. That's our key advantage, is the ability to recognize patterns, whether they're there or not, you know, the signal or the noise. But what's your opinion on this question of intelligence? What does that mean?

Can machines be intelligent? And when you hear the term artificial intelligence, which has scary connotations, and now there are companies like Open AI that want to make sure that artificial intelligence is responsibly designed and deployed. This is all kind of sci-fi scary stuff. So that's like five questions, but you know where I'm going. I'm just curious, curious your opinion on this big question of what is intelligence? Can machines have it and should we be worried? Well, let's see. How do we tackle that? I don't know the answer. I don't think anybody knows the answer. I think it's much easier to say what isn't intelligent than what is. I think it's much safer to say what isn't intelligent.

I think the New York Times article that we talked about a moment ago spends a lot of time saying what isn't an example of artificial intelligence in an effort to demonstrate perhaps that certain things are, in fact, artificially intelligent. Google Maps is a great example. The article talks about going back in time with an iPhone and having Google Maps on it and showing it to somebody to whom it would no doubt represent the pinnacle of artificial intelligence, right? And I think the article says something about it does something that a person could do in theory, but it does it so much faster, so much better, with such efficiency,

you could never hope to match it as a person. Therefore, it is artificially intelligent. Today, I don't know anyone who would think of Google Maps as being an example of artificial intelligence. And yet, if we take a step back, what does Google Maps do? Well, we give it a problem and it goes out with an infinite number of solutions, and it finds an answer. Is it the best answer? We have no way of knowing, but it is a very good answer. from an enormous set of possible answers. And I would argue that that is an example of machine learning. Now, I don't work on Google Maps.

I don't know behind the scenes how they've chosen to solve this problem, and it may not bear any of the hallmarks of what today we call machine learning. But to an outsider, it does the same thing. It takes a bunch of X's, it takes a bunch of Ys, it finds a relationship between the two, and it presents that model. Now, if we take a step past Google Maps and we look at Google Translate, is Google Translate intelligent? Well, there's a famous thought puzzle

called the Chinese Room. John Searle. John Searle. I'll say it for your listeners, but they may know, if you put someone in a room and you give them all the rules of translating from Chinese to English and vice versa,

and you start passing pieces of paper into that room and having this person execute whatever algorithm they have to divine to perfectly translate, and then they pass the paper out, can it be said that the room itself and the person in it is intelligent. Does it understand what it's doing? And by the way, does that understanding even matter for them for deciding that the room is intelligent? I think that to a

degree, as soon as we strip any of these algorithms to their bare essentials, which is to say, take the X's, take the Y, apply a bunch of rules, known or not, fuzzy or not, it strips some of that magic away. Intelligence almost is defined in some ways as this special magic Ghost in the Machine. Yeah, exactly. We don't know what it is. Arthur C. Clark said, any technology sufficiently advanced

will be indistinguishable for magic. And maybe it is the exercise of exposing these magic tricks that, I don't know, cheapens to some degree what we view as intelligent. Another classic famous one is a Turing test, right?

Where effectively the machine needs to deceive, be able to deceive within five minutes or a series of text that came remember the exact original test, a human, that it is a human. And one of the great lines in the New York Times piece was this idea, to your point about Google Maps, that the goalposts are constantly moving out, that if you go

back 20 years, everything we have today looks like artificial intelligence, but that will probably be true 20 years from now. And the question is, how should people think about this in their working lives and their daily lives? I mean, the biggest question is replacement of functions. So everyone got worried in the industrial revolution that, you know, technology is a lever, right? It lets you do more with less. A loom or a machine can do a lot more than an individual can do. And now there's, I've heard like Kevin Kelly, for example, the guy who used to be the editor at Wired, talk about equating like artificial intelligences, like units of it to like units of horsepower in your car. Like in the future, you're just going to have these generally deployable

units of intelligence to work for you. So where does that leave people? Like I think let's talk about markets, for example, right? So you have firms which are black boxes, for lack of a better term. Renaissance technology technologies maybe D.E. Shaw, maybe two sigma, to name, you know, three famous ones. How much is just going to be handed over and what will be left for us to do? If translate, if maps, if all of these functions, it's so clear that the simple little machine does it way better than a massive group of humans could do. it. What's left? Tough to say, isn't it? I could take the cop-out answer, but I'll try and give you a real one. For a long time, I've had this mantra that models are a tool. In fact, models are just a tool. And machine learning models are no different. They're magical to some degree. They're a little bit harder to interpret and understand, but they are still just tools for people to use. So the real question is, to what degree is a person replaceable by a tool? Let's table for a moment to discuss. of some artificially intelligent robot walking up and just, you know,

willy-nilly going off and taking people's jobs. Let's talk about tools because that's what we know we can build and we can use. So if you're a financial advisor and most of what you do is based on experience that you've gathered and essentially mapping people in their utility curves to portfolios that are appropriate for them, you are sort of fulfilling a role that looks, awful lot like a tool and a role that therefore can be replaced easily. Historically, you mentioned a loom. It's a great example. You were doing something that was somewhat repetitive, needed some ingenuity, some creativity, but at the end of the day, was taking X and turning it

into Y. I think at some level, everyone's job looks a little bit like that. And for that reason, I think that it's very easy for everyone to imagine how these tools will help them to the degree that their job resembles this characteristic that I'm describing. Will people be absolutely replaced by these AIs to give them this glorified term

that I'm not even sure I agree with? Yes, of course. Productivity and technological advancement are always going to replace certain elements of our working structure. Will they in turn create more jobs? Yes. On part with the number they replace?

No, probably not. I don't know. I guess it's a most honest answer. I think it's very hard for me to imagine someone who lives in such a special place that nothing they do is in some way based on the gathering of experience and from that experience a fairly rule-based answer, even if they're not entirely sure what those rules are. That's where the machine learning tools come into play, where what you do is repetitive but not easy to define. Translating is a great example of that. Translating, in theory, is extremely straightforward. I have English, I have French, they mean the same thing,

therefore translating is going from one to the other. But if you sit down and try and write the rules for it, it's very hard. So perfect for these machine learning tools that we don't need that closed form solution, that magical just catch-all answer. But we know how to define this problem, and we know how to take steps towards the right answer. And as Google has demonstrated, you can do a very, very good job. So it's hard to imagine somebody who lives in a world where nothing,

they do can be, one might say helped, one might say replaced by a tool designed to do exactly that thing. But it's a big, big, big leap to then say that that tool is intelligent any more than it is to say that a loom is intelligent, just because it does something with greater speed, efficiency, and potentially accuracy than the person who used to do it. Seems like, again, it's language is limiting, right? But it seems like it might be useful to parse intelligent with aware, because maybe they are intelligent in the sense that they're able to, with given inputs, very efficiently produce desired outputs. And maybe what's left is if these tools allow us to better answer questions, you still have to come up with the questions. That still seems firmly to be

the domain of humans and aware humans, conscious humans. So maybe one answer to the question is that you need to be thinking about the right questions to ask, not how to find solutions. Because once you have the question, what all of these tools are allowing us to do is get to answers more quickly. And the other thing I think about is relationships. Like you try to find things that are, I think, always going to be necessary. Now, maybe, like if anyone's seen that movie, Her, I liked that movie, where you think that there's a basic need for relationship and you could always do well if you were good at cultivating relationships. But here's this guy whose main relationship is with a artificial intelligence, I guess,

that is sufficiently advanced to basically pass the Turing test for him. So it scares me because the more I think about this, thinking about those goalposts constantly getting pushed out. Like 10 years ago, we probably would have said, wow, you know, lawyers and doctors, like those are esteemed jobs. A radiologist gets paid a lot for good reason because it takes tremendous experience. experience and now machines are actually better at doing the radiologist job than the radiologist. And that happened fast, really fast.

And this stuff is not on a linear, it's not linear, it's exponential. So I just get scared, frankly, that, like, to try to anticipate and cultivate the right skills as a working person that, you know, wants to learn and do well, to try to focus on things that if you're not the one designing these machines, which is really lucrative right now, you know, you might get paid high six figures coming out of college if you're really good at this sort of thing. But there aren't a lot of those people. There aren't a lot of those jobs, but there aren't a lot of those people. So it's just a fascinating open question of, and I've been rambling now, but maybe this dichotomy of intelligence versus awareness can get us away from

some of this problem. Yeah, I think the word intelligence is a dangerous one. I don't think anything that we call an AI today is intelligent. I don't think it's even close. It's a useful construct because it's a historically relevant construct. We've been calling things AIs for decades, always with the intent of building them, never having actually done it.

Now all of a sudden in the last decade with these enormous advancements in the state of the art for these machine learning techniques, we have things that we used to say were only possible with AIs, and now we do them regularly, translating. We keep coming back to that,

because it's such an incredible achievement that people could work their whole lives to be expert translators in a small number of languages, and now we have machine learning systems that do it for every language. Google recently announced that they are now able to translate between two languages without the Rosetta Stone that links the two by going through a third language,

or fourth even, and they still have these remarkably accurate, realistic interpretation, excuse me, translations of the source text. I think that intelligence, I don't know what intelligence is. I keep coming back to that. I always ask myself, what does it mean to be intelligent so that I can even start to say, what does it mean to be artificially intelligent? How do I bestow intelligence upon something?

I have no idea. So how can you say then that the things that we have now are not intelligence? If you can't define what my next question was going to be, so what would a thing have to look like for you to say, Okay, yeah, that's artificial intelligence. Gosh, I guess the answer you don't know. I don't know. I think we like to believe that there are things that we do

that are the consequence of intelligence. So gathering experience, let's go back to doctors and lawyers a second, we believe that those jobs are protected from the automation revolution because they require massive and deeply intellectual experience, gathering, knowing how to relate one thing to another. Well, along comes machine learning. which can gather experience faster than any person and at scale and benefit from the experience of its neighbor, right? It doesn't have to do it itself. We have these distributed networks.

So all of a sudden, gathering experience is no longer a qualification for intelligence, or at least not as I define it, because I can set up a set of a system of rules and have it go out and gather experience and tell me when a cat is in a picture as opposed to a dog and maybe what species of cat it is. Those are things that would blow anyone's mind very recently. And now we take them for granted. That's how we automatically tag images and videos. Caption them. So it used to, the goalposts are moving not only for the AIs themselves, but for our own definition of intelligence. It's not experience. Is it decision making? I'm not sure if it's decision making. I want to tell you that it has something to do with these leaps of

intuition, absolutely jumping from one area to another. But I'm sure if we had a machine learning researcher here with us, he would say, well, no, no, no, I train a model on cats and with some very small tweaks, it works wonderfully for dogs or it works wonderfully for driving cars. I don't know. Think about it in terms of something, you know, some huge revelation, you know, the discovery of relativity or, you know, Newton's discoveries or some major landmark historical finding. And I think, is that intelligent, could we think about intelligence in those terms? Like, could if a machine could somehow come up with a unified field theory, or if a machine is the one that does that, if such a thing exists, does that then make it intelligence? Like, could that be a definition

that, you know, we hold up a small handful of discoveries, you know, the origin of species, just there's three examples. We, there's a, there's a, we can probably name most of them if we sat here and thought for 10 minutes, maybe there's no better, if those aren't intelligence, who knows what is? Well, what if we said that that machine came up with that by brute force, by trying the infinite number of possible theories until it found the one that worked? So then it sounds like intelligence must incorporate some sort of efficiency or, like you mentioned optimization earlier, where the trouble with brute force is it's just time and iterations.

and what Einstein did maybe was incredibly efficiently realized something based on the inputs that he had gotten. And so he was just a much more efficient version of the same thing of pattern recognition. I mean, it has to come down to pattern recognition, right? Well, I think that we're very focused on intelligence and especially artificial intelligence as measured by what it does. And perhaps really the metric we should be looking at is how it does it. So this optimization, as opposed to brute force, as opposed to evolution, or rather evolutionary algorithms, I don't think that's how we do it, but I don't know how we do it, right? We know at a very physical, tangible level, we know how the brain works in the sense of neurons that spike and communicate with other neurons, but we don't know how that leads to consciousness or the thing we call consciousness if we really want to get sort of philosophical about this thing. tracks.

I mean, this conversation can go deep, but I think at the end of the day it comes down to a process. So I would be very disappointed to learn that all I do is take an exes and spit out wise with some well-known, well-defined optimization system. Perhaps the fact that my brain is a little bit random makes me intelligent or makes me think I'm intelligent. I don't know. It's a very scary road to go down because I think the conclusion that we ultimately draw is either we don't know or what we think we know can ultimately be boiled down to a system of rules which in theory any computer can replicate. So most people listening are growth-oriented people, meaning people that really want to learn and create new skill sets for themselves, explore, things like that. What can we learn from findings or maybe your own findings, Metlo and Data Company or elsewhere?

What can we learn from those ideas that we can apply to our own learning? So we're obviously less efficient than Google Maps, but are there like techniques that we could improve the way or the speed with which or the magnitude with which we're able to learn and improve ourselves? So unfortunately, for better or for worse, so maybe not so unfortunately, we don't learn like most machine learning algorithms learn explicitly. But in a reverse kind of way, the machine learning researchers have looked at how we do learn and try to encode that into the algorithms they build.

What's an example of that? So a little bit of stress turns out to be a healthy thing. If you just give perfectly clean data to a machine learning algorithm, you had to learn your data, but it has a difficult time generalizing because it's never had to in its training process. Whereas if you add a little bit of noise to that data, there's a popular technique right now, which is called dropout, which means that randomly you essentially break your model at random and you force it to solve the problem maybe a slightly different way or a slightly suboptimal way. The net result is a much more robust, much more generalizable model. And I think that that's one area where people could benefit.

Now, I don't know if the research that led to that was inspired by something that they observed in human, or not. But personally, I know that taking a break, working on something else, can sometimes get you, you know, thinking about an issue in a different way from a different angle. A change of perspective is valuable. Sometimes we get stuck in ruts. Ruts are dangerous, whether your machine or human. They're a little bit easier to diagnose when you're a machine. So like almost like an introduction of, you could do it two ways, like a break or actually stress, thinking about it the other way. Like if you're going, firing all cylinders, maybe a break is helpful. And if you're in a rut maybe or just have have leveled off or something in your learning, then stress, you know,

a mountain in front of you or something like that makes the speed with which you'll, you'll climb it faster. Yeah, or perhaps it makes you weigh the different routes you could take a little bit differently. So you have a whole bunch of ways you're considering to solve a problem and maybe stress makes you value the faster one as opposed to the more complete one, whereas if you had all the time in the world, you might take the scenic route. Right. What was the most interesting thing that you learned in your time at running Lowen Data?

I'll tell you the thing that led me to start Lowen Data, which was I read a paper by Jeff Hinton, who is mentioned many times in the New York Times article and is in many ways considered a father figure of the machine learning movement for decades. He, for many years, wrote about a class of machines called Restricted Boltzman Machines, and the details aren't so interesting. Sounds simple. Yeah, and unfortunately they are not. We don't need to go in the details here, but one of the interesting things about these machines

is that they could dream. Today we'd call that a generative model in the machine learning literature, but at the time there wasn't a good name for it, so they said the machines would dream. And what that meant is these algorithms were trained on handwriting, and by prompting the machine with a little bit of random noise, you could get it to essentially hallucinate

all the things it had learned. So up on the screen would come a seven, and then the seven would morph into a handwritten nine, and then that would morph into a one, and the one would change into a two. And the interesting thing was these smooth transitions between the numbers where the machine had never seen these transitions before. It had in some cases never seen the number written exactly like that before. But somewhere deep encoded in its memory or its parameters were these representations of these numbers and how to move from one to the other. And I remember reading this paper.

It must have been in 2009, 2010, and just falling in love with this idea that here is a machine that could dream. Now, at the time, I was convinced that was intelligence. But very much in keeping with our conversation today, I now no longer think that at all. I think that's a very deterministic outcome of the way that model was constructed. And today we have models that go significantly beyond what that model was and is capable. of doing. But that for me was this sort of aha moment of, wow, there is this remarkable thing happening. I don't even know that if I said machine learning at the time, six people on Earth would know what that meant. Now, of course, everyone seems to have some opinion on it. But it was very early days. It was

very exciting. And that was when I said, I need to go learn more about this. And Loan Data didn't start because I had a product or a business or a customer. Lowen Data started because I had a passion. and no real outlet to explore it short of going out and learning what problems people had and trying to apply these brand new technologies to them. So prior to that, you worked in risk management at a pretty well-known hedge fund in New York and now work for a private family office also in risk management. So your heaviest period of working as a data scientist is bookended by two periods in finance, specifically in risk.

And this is an area that I think people will be really fascinated to hear what you have to say because it is a gray area, to say the least. I mean, just the basic definition of it. I actually asked people to give me their favorite definitions. So I'm going to read them off because they're kind of funny ones, but some good ones too. So the most common you hear kind of refrains of defining risk is the potential for permanent loss of capital. in the finance context. Howard Marks' idea that more things can happen that will happen. Someone said base jumping, which is a good one.

Unknown unknowns, intentional exposure to uncertainty. What's left over when you've thought of everything else? And that's just a small handful of responses. And all are kind of interesting and good takes on risk. So we'll start with what is your working definition of risk? and then we'll go from there. Sure. So being a risk manager to me basically means being a professional skeptic. That's how I sum it up.

My job as a risk manager is to understand how things behave. To me, there's a very clear parallel between building and talking about machine learning and building and talking about risk management models. At the end of the day, I want to know how things really behave, not how they behaved historically, which especially in finance we can view is just one, draw out of many. And I do agree with many of the, many of the descriptions that you were able to pull up. I think all of them have some truth and none of them capture the whole of it. And I don't have one that does. As a risk manager, the analogy that I would use is trying to identify a sculpture in a

dark room. And all I can do is I have this flashlight and I can shine the flashlight on it, but I can't look at the sculpture. I can just look at its shadow, these projections on the wall. And I'm trying in my head to build this model of what that sculpture looks like. Because ultimately the questions in this context about a portfolio, presumably, are, well, what happens when oil moves such an amount? What happens when the market moves such an amount? It's not always about loss. It's not always about what happens when the market crashes. Those questions tend to come up in a risk context simply because they represent the absence of return.

which we always view as sort of the yin and the yang, risk and return, right? So when there's no return, we must be dealing with risk. But to me, risk is a much more nebulous idea, and return is the draw from the distribution that risk is trying to model. So ultimately, it comes down to understanding and defining the behavior of, in this case, assets and portfolios,

in all ways, up and down. One way I like to think about it is that any investing, strategy or process is seeking something specific out. And the common word for this is edge, you know, a source of alpha, some advantage that you have in the market. And obviously you want to build a portfolio that gains significant exposure to that edge. You could argue that beyond that edge, or if you were just to naively build the most advantaged portfolio that you could, there's almost always you drag in risks that you didn't intend. And now, risks typically are relative. So you're exposed to more energy stocks than the S&P 500. That's a risk. Or could be a source of

differences of return, right? It could be currencies. It could be commodities. It could be a million things. But that risk management may be one way of thinking about it is trying to identify here's your edge. What what risks does that pull in as a byproduct and can your edge survive neutralizing those risks? So can you you know maybe your process tends to make you buy all E&P stocks can you get the same exposures to your skill by by diversifying across all 10 sectors? And if you can then that's like effective risk management. Is that a fair way of thinking about Absolutely. I think one of the important sort of tenets of risk management is that we can't, or rather, in 99% of situations, we can't just get rid of risk. We can't wish it away. If we imagine the most simple example, we just have a bell curve and risk, we're going to just say that risk exists in the left tail of that curve. I can't just wish that tail away. I can pay somebody to take it away. I can allocate my portfolio such that it looks a little bit different. In other words, I can shift the mass from that tail. elsewhere in my portfolio, but I can't just get rid of it. It doesn't work like that, at least not

without some cost. And a big part of risk management, therefore, is understanding how to shift the distribution around. Okay, so you tell me you don't like your exposure to energy stocks. Well, that's fine. We can move that exposure wherever we want. We can pay someone to get rid of it. We can replace it. We can offset it. But it's not going to vanish. It's going to mutate and we want to make sure that when it mutates we know where it goes and we are we are still quantifying it elsewhere in that portfolio so risk can broadly be split into risk measurement and then risk management risk measurement is somewhat easier it it's also where a lot of folks will just stop because right I want to measure my risk well we have this even more

philosophical question of what is risk in the first place right and of course my answer there's a lot like my answer for what is intelligence well i don't really know but it has something to do with behavior and and how things move over time and evolve okay so how am i going to measure that well let's say for argument's sake that i think risk is um volatility and i want to minimize that well that's very easy i just go to cash and i have no volatility right so now we layer in um the edge that we think we have and we think that our edge is in energy stocks okay so we want to portfolio that's not quite cash, but looks a lot like an energy portfolio. But maybe it's not just energy stocks. Maybe it's a certain type of energy stock or a certain sub-industry, a sub-sector

that we want to allocate towards. Well, that's great. So we're going to take all this information and we're going to build some portfolios. And now we're going to try and understand how do those portfolios behave. What does the distribution look like? Sometimes we can quantify that with publicly traded portfolios. It's relatively easy to quantify that if the metric we choose is observable, like historical volatility, for example. With some portfolios, especially as we get into derivatives, illiquid assets, private assets, quantifying risk becomes a lot harder. But discussing risk remains just as important.

So we come up with some way of identifying what we think is risky after we've layered in all these edges and our objective. Then the risk management starts. And risk management, in a simplest sense, is just keeping up on that portfolio, knowing where it is, knowing how it's behaving. I have yet to hear of the risk manager who ran in when all the sirens were going off and saved the day because he slammed his hand on the red button and the firm was saved. That doesn't happen, and it's a myth, and no one should expect it.

Being a risk manager can be a very thankless job. You either get it right and are never heard from, or you get it very wrong and everyone knows your name. So how do we reconcile that? Why would anyone choose to do that? Well, if risk becomes an information function, where understanding all possible outcomes, not just the bad ones, but the good ones, the average ones, the simple questions of, well, the Fed's going to raise rates, what does that mean for our portfolio? Might mean a good thing, right?

Might mean a gain and value. Might mean significant loss. Those are risk questions, to me. wrapped up often with a return kind of answer. But I view the job of a risk manager as truly understanding in and out how a portfolio behaves and how external forces and internal decisions will change that behavior. Is it done mostly because portfolios change? Obviously, you make new investments, especially in more sophisticated portfolio where there's

this problem of private, very illiquid, super wide distribution of outcome type investments, venture. seed investments, whatever the case may be. It seems like to be able to actually get your hands around all that in a changing portfolio, there needs to be like a quantitative model behind it. How much of this is purely quantitative versus qualitative? That's a wonderful question. I don't think I agree with you.

I don't think you need a quantitative model, though I have yet to meet someone who would actually put that into practice, and I certainly wouldn't. I think it's a remarkably helpful tool. I couldn't do my job well without it. But if I had to do one thing to implement a good risk management program, I would have lots of conversations. I think that, going back to our intelligence conversation,

I think that our brains are remarkable pattern recognizers. They integrate experience and information in really wonderful ways. And when you're dealing with people who have had a lot of success, especially in finance where so much of the information you see is purely random and restraint plays a big role, right? Not getting fooled by some spurious correlation or random outlier observation into thinking that that was a signal. That was just noise. That was an outlier. We can disregard it. You sit down with someone like that and you talk about outcomes. And you're trying to sort of prime the pump.

You're trying to get that person, that portfolio, that asset, that algorithm. ready for whatever may come. And risk measurement is much easier with quantitative tools, because risk measurement tends to be backwards looking. It tends to be data driven. It tends to be empirical. But management, short of purely quantitative portfolios, which, of course, would faint if I said,

no, no, no, no, go talk to the algorithm. Go have a conversation and figure things out. To me, that's the most valuable thing that I can do, is trying to communicate qualitatively what these outcomes might be. Because, you know, this is a little bit experience-driven for me, but I worked with human traders, which meant they had to make decisions about how to allocate their portfolio. So if something surprising happened,

we may not want to run it through a model. In fact, it may be wrong-way risk in the sense that that might be the exact moment that our model is not well-suited to produce an answer. So instead we want that person whose remains the best pattern recognition system I'm aware of to be able to take this new information and actually come up with an intelligent answer. We don't want this to be the first time they've considered this outcome. So we walk through it.

We do fire drills. We talk about it. We say, what could happen? And I think that approach has been, for me personally, phenomenally valuable. There are enough people doing enough different things that there's no one-size-fits-all here. But if I had to choose one thing and if I were in a firm where it were something that makes sense, I would have conversations all day. So it sounds almost like the skill or the talent is to rapidly, you said to me once, one of our lunches years ago, we were talking about like if you had to isolate your core talent what it was. And you said, I'm very good at separating signal and noise.

and another way of saying that is uncovering these unknown unknowns rapidly, to use Rumsfeld's term. And there's this idea that right now the biggest risks in the market are the ones literally nobody is talking about, or maybe a small handful, you know, the guys in the big short or the tiny minority that we're talking and thinking about it. Is that fair? Is it people that would the best risk? Because you think of risk manager, frankly, as somebody that is trying to suppress exposures

or make exposures look more like the thing you are trying to beat. That thing could be the S&P 500. It could be 0%. Whatever the threshold is. That's how I think about risk management, reducing those chances or suppressing those exposures. But it sounds like what you do is much more hands-on. I'd like to hear more about it. So is it as specific as you work for family office?

They have relationships with investors where they are giving money to investors to invest on their behalf. Are you literally talking to the people behind those investment processes and effectively being like a, like you said, professional skeptic stress tester and just asking them questions to get them outside their comfort zone? There's an element of that. I mean, listen, at the end of the day, the secret to knowing the difference between signal and noise is just to, assume everything is noise. That's where the skepticism comes into play. It's not that I have some magical toolkit or deep insight and I can say, oh, well, this has meaning and this doesn't. It's that I literally sit here and I just assume everything is random, everything is noise. And if you want me to believe otherwise, you have to prove it to me. But until you do that, I'm going to sit here

and I'm going to build models of what I think the world looks like. And to the extent that we deviate from that, I will be aware of that. I will know that. Perhaps it's information. Perhaps it's not. But that null hypothesis, if you will, is a powerful one. And it's a very, very easy one to deviate from because the second you take one step into believing that your edge is both real and manifests in in the data and noticeable ways, it's now very hard to go back to the world of skepticism. And everything is noise. And I don't take on blind faith that things work. So a good example is when you talk to a quant manager. So a quant manager will never tell you how their algorithm works, obviously. That's their, that's everything, right? That's their IP. Also, even if they did,

you might not understand, right? It could be a machine learning algorithm that they've chosen to implement in a very specific way. And you know what? Maybe it's not a machine learning algorithm. Maybe it's an evolutionary algorithm. And even they aren't sure exactly how they came to this answer, right? They can't trace the math exactly. They've wound up with a very powerful model that seems to pick stocks amazingly, and their returns in their back test are wonderful. Now, I've never seen a bad back test. I don't think you've ever seen a bad back test. I've seen plenty of bad back tests. Then you're much more honest than many of your peers. Well, there you go. Yeah. My back tests are always terrible. Somehow they're just never as good as what I'm trying to match up to, right? So how do you get information? How do you get information? out of such a closed system. And again, all we can do is poke it. And we're just looking for reactions that are a little bit odd, that lead us to question

the claims that this person, that this investor is making, or conversely, lead us to abandon our own null hypothesis that the world is random and you have nothing to offer me. So tell me about the toolkit for doing that. So let's say all you've got is a return stream, which is probably the case for, certainly like a completely closed black box type quantitative approach or other black box type approach. You might just have monthly returns or daily returns. So how do you go from that? How do you use that to poke? What are the tools you use? So we have to get away from the monthly returns as quickly as possible for the for the simplest reason that especially if they come from a

back test. Forget back. Let's say it's live. So it's live. So our job now is to understand the distribution that gave rise to those returns. So we understand that those returns are draws from a distribution. What is that distribution? What do the tails of that distribution look like? Is it a very, very peak distribution with a very tight center, but then it has this enormous left tail? Because you've been, I don't know, selling puts all day. Is it very well behaved? Does it change frequently? Is it stationary? We're trying to get answers to these questions. Nobody knows the answer to these questions, which makes it very difficult to answer them. But again, we go back to that metaphor of shining the flashlight in the dark room. We're just trying to get flashes of an idea. We'll never

know for sure. Again, this is very much art rather than science. It's very qualitative as opposed to quantitative. We're trying to gather evidence. And this is not a gotcha game, right? We're not trying to make someone uncomfortable. We're not trying to push them out of their comfort zone. It's not like if they take a step too far and the whole sham will be revealed to us because in many cases, these are not just legitimate, but extraordinary strategies that are being put into place and that you would absolutely like to be an investor in. The goal instead is to understand them. Now, if in the process you reveal the whole thing to be a sham, then more power to you. You've dodged a bullet. But I always try to you.

to imagine that I am going to have to communicate this to yet another third party. Am I able to adequately do that in a way that doesn't simply refer them to a tear sheet of monthly returns? Can I explain what is happening? I also have to believe that what I'm explaining is not some sham perpetrated just for me, right? That I legitimately am having some window into this strategy and how it works. and that's my objective. So if asking tough questions about uncomfortable situations is how we get there, then so be it. That's how we get there.

And if we have just a very nice conversation about the philosophy of the investment strategy, that's another way to get there. And I'm happy to say I've had as many conversations of that type as the other. In the two experiences you've had with hands-on direct risk management, to what extent, like what magnitude and how often did your involvement, change what the portfolio would have looked like had you just evaporated and not been there? I'd love to tell you it's just constantly like that, but of course that's not the case. There have been times when my involvement has clearly directed the course of something.

I won't go into specifics now. But more broadly than that, I think my contribution has been building tools that help turn these very qualitative ideas we've been talking about into something tangible. One of the most effective tools that I've put together has, though it's been driven by numbers in the back, has had no numbers at all in terms of its presentation. And that's been a graph of exposures, a 3D graph of exposures. And by having many conversations with this chart next to me and seeing it evolve through time, we got to a place where the folks I was working with could look at this chart and intuitively understand how they were positioned. You could look at this thing and you could say, oh, I've got tail hedges in place, or I'm very exposed to such and such an event.

It's a little bit difficult to describe this thing without actually showing you. But through this graphical techniques, we were able to communicate what was really driven by a thousand different numbers, but no one can look at a thousand different numbers and actually understand what they are, but we're very good at looking at pictures. So we turn those numbers into a picture and then anyone can glance at it and visually taking this information and make decisions based on it. And I'd like to think that throughout my career there have been many, many times when through the use of these tools and by demonstrating possibilities to people and by trying to present the behavior of their portfolio,

I've had some meaningful impact, if not on the course of the investment, than on the decision to pursue it. In the 3D graph, which I want to see, I haven't seen. What are the dimensions? What, what, what, what, what are you plotting? Sometimes. What are the exposures, I guess? Yes. Sometimes, you know, the challenge of visualizing something is that, of course, these portfolios existed many, many dimensions and very high dimensional space. So, visualizing that can be a challenge. So sometimes we do it statistically. We choose, for example, you could use principal components and just say, what are the two

drivers of returns in this portfolio? And given how many options, like how many would that have to choose? Like you could have equity beta in there, you could have oil, you could have. Is that, am I thinking about this sort of way? Yeah. So if we were going to choose some, we would probably just go out and say, well, what do we think is representative? Well, this portfolio is highly exposed to oil and credit. So let's make a graph of oil exposure, credit exposure, and.

Chartered. Yeah. What's our exposure? And look at it. If we were going to do this with the more statistical approach, we'd go out and we'd say, well, we have no prior belief about what we're exposed to. Let's just let the system go out and find correlates and plot them. The important thing here, ironically, is not to deliver necessarily useful quantitative information,

but rather to build something that someone can look at and intuitively have some understanding of not what their portfolio looks like at any moment, but how it's evolving. That's an important point. I don't think I mentioned a moment ago. The key thing here wasn't that you would know where you were to, today, it was that you would see how you were evolving over time. Now, I didn't know at the outset of this that that would be the important thing, but that's what I learned. That's something that I took away from this, from building these things, is that the evolution of the portfolio turned out

to be just as meaningful as its current position, because sometimes you meant to change it, and sometimes it changed as a consequence of external events. And that may have been sort of a frog-boiling situation where it just was slow and you didn't notice, and that may have been something that just happened overnight and, you know, snuck in because it's not a principal risk that you believe you're exposed to. You wouldn't go out and measure it every day. That makes me think back to the beginning of this part of the conversation

of this idea of intentionality. That was one of the definitions where risk, maybe one way of thinking about it is, and this idea of change over time is interesting, that every process has some byproducts, and this is a way of identifying, like, creep, where you did, you, you, didn't intend to have these exposures, but they've crept in over time. And maybe risk management is about first, like you said, measuring, identifying what those are along a number of different

dimensions, and then somehow systematically snuffing them out, or at least being aware of them when you're making future investment decisions. Yeah. And we have the unknown, unknown problem there, right? Because your dimensions don't contain probably the most important one. Right. We don't know what snuck into this portfolio. And even if we did, we wouldn't necessarily know what to comp it, too. Yeah. But that's why doing this on a constant basis and reviewing it, even when you think you understand it, to me, it's just critical. Now, all of this, I'm sure, has some out there thinking, you know, the complicated nature of all this sure makes a Vanguard, you know, 60, 40 portfolio sound pretty darn attractive. What is your opinion?

I've actually tried to steer clear of the active, passive index debate with most people on the show, just because frankly I'm bored of it. But you have a totally different take. So I'm curious where you fall on the spectrum here. And I'm going to assume you believe in one component of this, which is that low cost is better than high cost. But maybe I won't even assume that. That's a very fair assumption. Where do you fall on the, I guess, now you're a manelbrot fan. So I think I know part of the answer. But in this active, passive debate, what are your thoughts? a very interesting conversation a few weeks ago. I taught a class at the Booth School at the University of Chicago. And we got into a little bit of a debate along these lines. Was it the quantum mental

class? It was the quantum mental class. It's really wonderful. I wish I had had the chance to take it when I was in school. That name is like birth control. Oh, probably. Just tell your wife you're taking a quantum mental class. You should take care of it. That's it. We got into a little bit of a debate about whether there is such a thing is passive investing. And whether what we call passive investing today is in fact expressing a relatively active view in that you have to go out and choose how you want to index. Do you want equity exposure? Are you basically taking on equity risk premia and just sort of absolving yourself of the need to rebalance and choose stocks? That's an active view. It's active in that it has an absence of

some actions that other strategies do. But by that definition, we'd say that some of the largest hedgerons in the world are passive because they're not high frequency. They haven't met some minimum threshold of activity to be considered active. So a little bit that's semantics, but it really got us all talking about whether or not stepping away was the same thing as not being active. And where that conversation ended up going was to say that we often pay people to be active when in fact they're adding no value, which is not to say that the opposite of that is to be passive. It's simply to be active at a much lesser rate.

And I come out increasingly, even in the last year or two, much more on what we would call the passive side or the less active side of things, simply because I don't see on average the returns to active investing. There will always be active investors, and some of them will be quite good. But if you asked me on average, I don't think simply going out and trading and expressing an edge and having a view means that profits occurred to you on average. Why has that, you said last year or two?

So we've come through a period now where when you look and match it up historically, it's one of the, I don't know if it's the most, but one of the most impressive periods for, say, market cap weighted indexes relative to active. managers. It's been a dismal period for active management. So what what drives the change in your opinion of the last two years? Is it just the lack of active success? Because we know that is cyclical to some degree. Now long-term active aggregate has to lose because of costs, but it can go long periods where it works very well. So what's driving that change of opinion? That's a great question. It would be very dangerous for me to start espousing an opinion that was short-term driven, especially, as you've pointed out, because the cycle is pointing one way and not the other at this moment. It means it's probably actually the exact wrong time to do what I've just described.

But to me, one of the interesting things is the rise of the Robo Advisor. So passive investing, I'm making air quotes. I just realize no one can see that. But air quotes, passive investing, until recently basically meant index and forget, right? Set it, forget it, go back to your business. Rebalancing, too bad. You didn't think about it in time and you only thought about it because the market crash and now you want to come back to it. But the rise of the robo advisor means that there's this very interesting middle ground, which, again, I think is far from truly passive because within all these robo advisors,

you're going to dial in your level of risk. In some of them, you can choose certain factor exposures or perhaps not factors, but, you know, tilts that you'd like to have in your portfolio. And what I think that they do is allow anyone to set up the portfolio that they believe to be most appropriate for them. Now, the goal of that portfolio may not be to outperform because we're kidding ourselves if we think we can all, you know, have an above average return, right? Chances are, through action, we're going to have a below average return. That's how a small number of people are able to accrue.

wealth in this model. But to me, it's not the cyclical indicator that, oh, indexes have done well, therefore we should all index. In fact, it's the opportunity to leverage these relatively new tools to implement a strategy that would have been much harder for people to implement in the past and would have incurred much more of a cost except in the last few years. You get a window into a lot of hedge fund managers having worked there, but also allocated to them, understood their processes. Is there anything on that side, the qualitative

side of the equation, or from that group specifically? So you see, obviously, it's been a tough time for, say, long short equity funds. There's this concept that everyone's talking about with the paradox of skill where absolute skills at an all-time high, but for Al-Fall, that matters as relative skill, which is at an all-time low. If you look at things like sharp ratios or whatever, have you noticed anything kind of from the inside looking at hedge fund managers being a part of hedge funds that has changed that that also colors this change of this view that a more passive approach might make more sense going forward? I think that from the perspective of a U.S. taxpaying investor, you're no longer the prime target of a hedge fund manager, whereas you were

10 years ago, 20 years ago. The incentives have shifted such that the marginal buyer of a hedge fund is now a pension fund, a sovereign fund, folks who have different after tax, after fees, after liquidity requirements than an individual or a wealthy individual or family office, for example. I think that's led to behavior shifts in how hedge fund managers run their own operations. They want to cater to this this investor who ultimately drives flows at this time. So at a very high level, I observe some behavioral consequences. At a much lower level, it's also much easier to start a hedge fund, right? So there's a lot more of them because they're taking advantage of low-cost platforms,

quantitative access to markets. There's a lot more to sift through. That's not to say that there's still as much quality out there. but you got to wade through quite a bit to find it. And that effort has some cost in addition to what you'll pay to the hedge fund itself. I go back to this sort of fallback attitude of skepticism. I don't need to go out and find the greatest investment. I can sit here happily and assume the world is random and noise and there's nothing to do.

But I want to be convinced, right? I want to believe. And, you know, I'm happy to say that there are many, many wonderful investment managers out there who deliver real returns in attractive and appropriately risk measured strategies. But the activity of finding them can be difficult and can be increasingly difficult as this becomes an industry that is increasingly broad and splintered. I don't have an answer to that challenge. I'm no longer on that side of the equation.

and I haven't frankly spent a lot of time thinking about that. But this is, it's certainly an evolving industry. I have the luxury of being able to throw up my hands and wait and not have to make a decision, which I think is more of a luxury than it sounds, to be honest. Do you think that coming back to full circle to machine learning and artificial intelligence, I'll keep using that term even though you don't like,

like it. Well, we can't get away from it. We might as well use it. Whether those will, given how fast they are evolving and given that the world of investing is is ripe for applying these kinds of tools because ultimately everyone is doing the same thing. They're collecting data. They're interpreting or changing that data somehow using a proprietary process or filter, experience, or model, whatever it might be. And then they're using that to build a construct a portfolio. So the broad process is the same. And that kind of staged process seems like, well, if you give it the right data and you tell it the right kind of thing, define the error terms and all these kinds of things, that the managers that use that technology will crush the ones that

don't. Unless, again, thinking about relative skill, like everyone's using the same thing. And then maybe the dummy that's not using anything might win. Well, there's a very dangerous consequence of what you just described, which is this. These machine learning models can have millions, even billions, let's just say, of parameters. They can fit any data you choose to show them. And they have become very accessible in the sense that I type two lines of code and I'm running, you know, a seven-layer-deep neural network and guess what? It's perfect. The back test is amazing. The danger of overfitting is high. In many ways,

sticking to the more traditional, let's call them, algorithms, the linear regressions of the world, the simple classifiers, it regularizes the model. And that's really important. That's really important in machine learning.

We spend a lot, a lot of time thinking about how do we prevent overfitting. I mentioned a technique called dropout earlier, which is just a... Scramble. Yeah, it's a drop-dead easy way to try and build a robust model.

sometimes it works. In fact, most of the time it works. We spend a lot of time doing that. One easy way to regularize your model is to use a simple model. Just don't let the model make a lot of choices. So linear regression, it's a close-form model, so it only has one outcome. But we could set up a model that effectively looks a lot like a linear regression, but we don't let it use the closed-form solution. We force it like any other machine learning model to iteratively approach that optimal solution. Well, that model is hamstrung a little bit by the fact that it's a very simple model. It only has as many parameters as we have regressers. So that could actually be an asset in helping practitioners avoid overfitting, which to me is the greatest danger today as folks go out into

this quantitative world. Overfitting can come from two places. It can come from the model itself. and it can come from a belief of the person who implemented the model. I get a great back test or even a great forward test, and I choose to believe that that is meaningful as opposed to random. So false confidence. False confidence in the output of the model is a form of overfitting. Fascinating stuff. It's going to be a, sure will be an interesting decade or two to watch.

Because, again, at the end of the day, these are businesses that people are buying and selling. And there's only so much that we can know about every individual business. And the competition to gain an edge at the bottom of the well, the minutiaa is intense and fascinating. So the last couple of questions for you, ones that I ask everyone. The first one is to hear the most memorable individual day of your career. That would have to be the day Lehman went bankrupt, especially as a risk manager.

It was kind of extraordinary. There was a special trading day. It was a Sunday when firms could trade to try and offset their credit exposure. And I believe the terms were that if Lehman filed for bankruptcy before midnight, the trades would stand. But if they didn't, the trades were all, you know, they'd all go away. And I remember there was just this enormous logistical operational challenge of we essentially were, going to try and run two sets of books, one with the trade standing and one without the trades, because we wanted to know our exposure in both events. There's also just the challenge

of mapping out what trades we chose to make or would choose to make, how they would impact our portfolio. It was just as sort of an extraordinary undertaking. And, you know, I was not new on the job at that point, but I was certainly still learning. And I was very fortunate to work with an incredible team and have an incredible mentor for a boss who really understood both the gravity of the situation but also the very pragmatic reality that we had to deal with. And I remember being there Sunday, we're turning through everything, we're getting the trades on, we're recording them by hand in many cases, just checking things off. I mean, the portfolio was extraordinary. And as it happened, as we all know, they did not file by midnight. The trades all disappeared. And I

remember sitting there and we waited and we waited and early Monday morning the call came and they were filing it was maybe I don't know three in the morning four in the morning something like that and all of a sudden this whole other contingency set of plans got kicked into action um and there have been a couple of of incidents that being one which have really informed my view of the risk manager's role and and why I care about it and you know in a lot of ways it sounds like one of these things where, wow, you really better love this or you're really going to hate it kind of thing. There's no, no one goes into this job and it's like, eh, all right, I'll do the risk thing.

It's not a good day job. Exactly. You really got to, you really have to care for one reason or another. And one of the reasons that I've come to care is because I've seen in a number of instances, what can happen if proper precaution isn't taken. Fascinating. You're the second person to say Lehman, which is, which is, I guess, not surprising. Well, it was such a momentous, yeah.

Yeah, I got a momentous day, at least in the last, you know, 15 years. The last question is what is the kindest thing that anyone's ever done for you? In my career? Interpret it however you want. I'll stick to my career so I don't embarrass anybody. I have benefited enormously from people's kindness throughout my career. I've had a strange career in the sense that I've never had a job that I could interview for. Well, I've interviewed for jobs, but the requirements of my job have always evolved in ways that I could

never have anticipated. And had I tried to interview for the job I later held, I'd probably be laughed out of the room, simply because it's somewhat R&D driven, it's somewhat fanciful, it's just evolved in strange ways. And part of how my career has gotten to those points is because people have put trust in me, but also encouraged me in surprising ways. I have I have, I have left jobs on great terms, on wonderful terms and maintained relationships. I have come to jobs simply because we mutually agreed that there was something interesting to do. And those are small forms of kindness. I could point to momentous examples of kindness in my career. I'm going to choose to keep them private. And instead I'll just characterize the whole thing as

as being very fortunate to work with people who cared about development, both personal and professional, in equal measure with the sort of basic profit maximizing objective that we all take on on a daily basis. Awesome. Well, this has been a blast, a bit of a mind scrambler, but I think the topics at hand require that. It's tough stuff. And if you're not one of the people, building it, it can be very intimidating and hard to understand. And even after a conversation like this, there are so many open questions about not only how this affects portfolios and risk, but how it affects jobs and our lives. We know that it will affect our lives in significant ways it already has. So it's been really fun doing this on tape instead of just casually. Maybe

it'll be the first of many, but thank you so much for your time. Thank you so much for having me. I've deeply appreciated being here. Thank you. Hey everyone, Patrick here again. To find more episodes of Invest like the Best, go to investorfieldguide.com forward slash podcast. If you're a book lover, you can also sign up for my book club at investorfield guide.com forward slash book club. After you sign up, you'll receive a full investor curriculum right away, and then three to four suggestions of new books every month. You can also follow me on Twitter at Patrick underscore Oshag. If you enjoy the show, please leave a quick.

review for us on iTunes, which will help more people discover Invest Like the Best. Thanks so much for listening.