Invest Like the Best with Patrick O'Shaughnessy - Richard Craib - Crowdsourcing Predictive Algorithms - [Invest Like the Best, EP.102]

Episode Date: September 4, 2018

I intentionally avoid the world of quantitative investing on this podcast. The whole point of this format is to learn about many different fields, and the vast majority of my time is already spent in quant world. Occasionally I’ve broken this rule because of something unique, including this week’s conversation with Richard Craib, the founder and CEO of Numerai. If you listen to the podcast often you’ll have heard me reference Numerai, a hedge fund which blends quant investing, cryptocurrencies, crowdsourcing, and machine learning — talk about a PR company’s dream. One important note: Numerai is both incredibly open and very secretive. You may sense a bit of frustration on my part, but that is only because, as a fellow quant who loves details about data and modeling, we couldn’t go deeper into the details on the record. We discuss how Numerai has created an incentive structure to work with data scientists around the world in an attempt to build better investing models. The idea of having data scientists stake cryptocurrency in support of the quality of their models is fascinating. Like many hedge funds, Numerai doesn’t share its track record, so we don’t know if this works—but I hope you, like me, use this conversation as inspiration for how different technologies can intersect. Hash Power is presented by Fidelity Investments Please enjoy my conversation with Richard Craib.   For more episodes go to InvestorFieldGuide.com/podcast. Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub. Follow Patrick on Twitter at @patrick_oshag   Show Notes 2:32 - (First Question) – How he came up with Numerai and how its related to his background 4:08 – How he works with and models the data for his system 5:24 – Describing machine learning as it relates to his work, and specifically linear regression 7:11 – The important stages in his sequence 8:46 – How the scale in the number of data scientists they use is different from other areas 11:30 – Which is the most important aspect of creating alpha; their data, algorithm work, proprietary ensembling of those algorithms. 14:30 – The idea of staking in blockchain 17:30 – Does the magnitude of the stake matter in blockchain 19:10 – Understanding the full incentive structure for both staked and unstaked work 21:07 – How is the prize pool determined 22:29 – Philosophy on how to source interesting data 26:11 – His thoughts on the crowd model and the wisdom of crowds 27:12 – The size of stakers for Numerai 27:51 – Interpreting the models and knowing when something is broken 30:03 – How they think about people not submitting their models 31:48 – Their model building 32:39 – Most interesting set of things they are working on to improve the overall process             35:38 – The Market for "Lemons": Quality Uncertainty and the Market Mechanism 37:11 – How people can come along with their own data 39:00 – His thoughts on the quantitative investment community 40:44 – What else is interesting him in the hedge fund world 44:03 – Building a marketplace and staving off competition 46:16 – Kindest thing anyone has done for him   Learn More For more episodes go to InvestorFieldGuide.com/podcast.  Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub Follow Patrick on twitter at @patrick_oshag

Hello and welcome, everyone. I'm Patrick O'Shaughnessy, and this is Invest like the Best. This show is an open-ended exploration of markets, ideas, methods, stories, and of strategies that will help you better invest both your time and your money. You can learn more and stay up to date at investorfield guide.com. Patrick O'Shaughnessy is the CEO of O'Shaughnessy Asset Management. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of O'Shaughnessy asset management. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of Oshonnessy asset management may maintain

positions and the securities discussed in this podcast. I intentionally avoid the world of quantitative investing on this podcast. The whole point of this format is to learn about many different fields and the vast majority of my time is already spent in the quant world. Occasionally I've broken this rule because of something unique, including this week's conversation with Richard Crape, the founder and CEO of Numeri. If you listen to the podcast often, you'll have heard me reference Numeri, a hedge fund which blends quant investing, cryptocurrencies, crowdsourcing, and machine learning. Talk about a PR company's dream.

One important note, Numeri is both incredibly open and very secretive. You may sense a bit of frustration on my part, but that is only because, as a fellow Quant who loves details about data and modeling, we couldn't go deeper into the details on the record. We discussed how Numeri has created an incentive structure to work with data scientists around the world in an attempt to build better investing models. The idea of having data scientists stake cryptocurrency in support of the quality of their

models is fascinating to me. Like many hedge funds, Numeri doesn't share its track record, so we don't know if this works. But I hope you, like me, use this conversation as inspiration for how different technologies can intersect. Like the Hash Power documentary, this episode and other Hash Power singles are brought to you by Fidelity Investments, a company that is constantly researching and experimenting with emerging technologies like crypto assets and blockchain to improve the lives of their customer. Fidelity provides a comprehensive set of products and services to individual investors, employers, and

financial advisory firms. For more information, please visit Fidelity.com. Please enjoy my conversation with Richard Crave. So Richard, thanks so much for joining me today. You represent such an interesting intersection, which is my favorite kind of investing conversation between kind of traditional quantitative investing, machine learning slash artificial intelligence, but also blockchain technology. And I've always used Numeri as one of the interesting examples of people using blockchain

for something that seems to be make a lot of sense. So maybe you could begin just by giving us a touch of background on how you came up with this idea, maybe how that's related to your own history and investing, your own professional background, and then we'll go as deep as possible into some of the interesting features of the Numeri model. Sure. Yeah. The first sort of ideas for Numeri came when I was working as a quant.

And there was some quant in the fund I was working at, but there wasn't any machine learning that they were doing. So I decided to spend a year recently. researching how we could use machine learning. And then I finally found something that I thought was quite good. But at the same time, I was also playing in data science competitions on Caggle. And I sort of thought, okay, well, the best way to get really good performance is to get the world to compete on the dataset. And that was true of nearly everything that had ever been done. The Netflix prize was won way better than the people at Netflix could do. And then every other

tournament on Kaggle, that was one, is like way better than the companies that were hosting the competition could do. And at this time, I was also reading about Ethereum. And I thought, you know, maybe there could be something interesting where finance, which has kind of stayed the same for a long time, could be changed by these two technologies, blockchain and machine learning. Maybe you could describe the actual process here behind outsourcing, if you will, the analytical or the model building piece. We're going to spend a lot of time on data. and what kind of data is and what you do to it to make this possible. But if you could just describe for those unfamiliar, the actual sequencing of what you put out there,

how you take that and incorporate it into your system. Well, we have the large data sets that is built up from different kinds of financial data. And it's structured in a way where you have these target variables. And you can use the data we give to make a model that predicts from the features to the target. And we decided to give away that. So no hedge funds really want to give away their data. But Numurai was the first hedge fund to give away all of its data for free. But all the data, it contains all the patterns of the stock market, but you have no idea what the data is.

So you have to use machine learning to basically find a way to model the data, even though you don't know what feature one means or feature two means. And that allows us to share our data sets with anybody without them being able to. to run off and start their own hedge funds with them, but they actually kind of are incentivized to come back and submit predictions to us, which we then trade. Maybe we could use this as an opportunity to kind of level set on some terminology and describe like a generic machine learning process as it relates back to maybe more traditional, like linear regression or things people might be a bit more familiar with. So you mentioned this idea of features and targets.

Maybe if we relate that back to what the actual data is that you upload. So when I looked at it this morning, you know, there's a whole bunch of identifiers that could be securities, they're sort of what I would call like blind factors. So something you've regularized, which is kind of numbers between zero and one, maybe you could describe kind of what that might represent. I would think about that as like independent variables. And then you've got outcome variables or targets or labels, whatever. But maybe if you could describe that generic process of machine learning as it relates to something more simple like linear regression, that would be a good jump off point for the rest of the conversation. Yeah. Well, machine learning is regression. And so it really isn't

much more than curve fitting to datasets. So one way to think about it is if you have a bunch of points in like a scatter plot on the X, Y axis, you can find a line that fits those points with a regression. But you actually don't need to know what the X and Y axes are to find that line. And on Numeri, there are 50 dimensions instead of two. I think it's 50 right now. And then the users find curves that fit those high dimensional spaces. And once you can have a regression, you have a model to predict something.

And so you're finding classes of points that should be labeled a target of zero, which might mean the stock is going to go down, or classes of points that are labeled one, which means stock's going to go up. And your curve that you find using machine learning to hopefully be very accurate at predicting new data as well. So I always think about these things, especially quantitative processes, as sort of all following a similar rubric and what's interesting about numerize, how differently it's doing certain parts of this. So first I'd be curious if you agree with this general framework, which would look something like engineering the features themselves, doing whatever

transforms you need to, in your case, regularizing them, selecting the features, the 50 you mentioned, in the case of machine learning, like hyperparameter tuning. And then you've got to, you know, select the models that are going to work that's being built by your audience, ensemble them maybe and put it into production. Do you think that that is kind of the, is that how you would describe the stages that are important in this sequence? Yeah, that sounds kind of right. I mean, so one thing that people can think about with something like Numeri is we are trying to crowdsource stock tips. But really, it's not like that at all. Like no one's telling us any specific stock to buy. We don't think it's, it makes sense to crowdsource that. It does

make sense to do machine learning competitions where you set problem up in a very specific way and you ask people to solve that. And so the regularization step, which is kind of like normalizing the data, making it seem like something that can be modeled, that doesn't seem too much like too non-stationary, then taking that is the part where we let our users. So our users aren't doing any insight into financial data. They just have to do the machine learning piece. And that's the important distinction with Numeri, where that's the piece that we want people to work on because that's the piece where the crowd is going to have a large edge over just one person doing it. Can you describe that edge in a little bit more detail?

So I feel like there are two key areas of edge. The first is the data itself. And one of the things we find is the more time you spend on getting unique or a really clean data set, you know, that drastically improves the outcomes of this exercise. And then the second would be the competition that you're running effectively in an open way, which is ever-increasing scale. skill at fitting that curve. Maybe just talk about the dimensions of that skill. Like, where is there diminishing returns? You know, if I'm a hedge fund and one of these huge hedge funds, quant hedge funds that has, I don't know, they'll call it 100 PhDs that are working on custom data set, why wouldn't there just be diminishing returns to the next 50 PhDs looking at the same thing?

There are only so many machine learning tactics. You know, it changes, but it's not, it's not that crazy. So where does that, where does that belief that going from 100 to however many data scientists you have in the world working on this competition, where? is that skill? Like what's being differentiated? There's definitely, you know, kind of only a handful of machine learning algorithms out there that people regularly use, like probably heard of neural nets and decision trees, random forests, support vector machines. There aren't that many of those. So it is true that anybody could kind of run through each of those and then just ensemble that. But really what you're trying to get at is much more creative approaches where there can be

certain kinds of emphasis on certain things that you wouldn't be able to do just by choosing a whole battery of machine learning tests. So some of what our users have done, they definitely are using a lot of neural nets, but some of what they've done is very peculiar. Like we can't tell exactly what they've done, but it ends up having really, really strong performance. So I think it definitely helps. And then it also helps in a technical sense. It always helps to have more. So if it is diminishing, it still helps to have more. But it wouldn't work if Numeri had to hire every single one of the, would be the biggest hedge fund in the world by far, by 100x,

if we hired everybody to actually sit in our office and work on this. So we can do things in a better, more efficient way by letting people do whatever they want, use whatever programming language they want, choose whatever algorithm they want, and that makes it very competitive and worthwhile to do it in this way. So the other thing is with financial data, if you have a 52% edge, to turn that into 53 has a really, really big impact on your sharp.

It's sort of like could be a difference of the sharp of 1 or 2. So that's why it's particularly helpful in this case. I think you can't really have a numerari for healthcare or enumerize for, you know, other kinds of data because they'd never have that kind of difference between a 52 and a 53% edge. So if there are kind of three parts of this creation of alpha, assuming that's the end goal here. The first being getting the data that you think matters. I want to talk about that.

The second piece being what you just described, which is this kind of global network of data scientists working on the best algorithms. And then the third being how you ensemble or create sort of a meta model, I guess, of the best models that people have submitted. Which of those three, or how would you assign the importance or the impact of those three categories on your outcomes, on the alpha itself, which is the most important by how much? Okay, wait, say them again. Which first?

So you've got three primary sources of edge, let's call it. The data that you have, what the data is, how you're collecting it, how you're normalizing it, you know, that's kind of proprietary. It's happening at Numeri. No one in the world knows what that data is doing. The algorithm work that the crowd is doing. And then the proprietary ensembleing of those models. I think that data is very important. I think the founders of 2 Sigma, I'm not sure if they said publicly, but they often say that it's the data. That's the most important thing. And that is kind of true. If we gave out a problem that didn't make sense in the first place, say it didn't account for the data we gave didn't account for costs or was tradable only in countries that you actually can't trade in the US by law, anything like that would be like the algorithms would then find inefficiencies, but that wouldn't really matter because you couldn't actually execute on that. So that's one part of the data.

And then the other part is having it be clean and very long. So I think data is critical. And Numeri doesn't allow our users to bring their own data. They have to use our data. And so we are limited by the data we have, but they can use any algorithm. So I think, yeah, data and algorithms are both good. If you have very bad data, it won't work. If you have a very bad algorithm, it won't work.

But I think the critical thing actually is not really in the list of three things, because the ensembleing of models, we don't use anything. We used to use a kind of very intricate way of ensembleing the models together. But since we released Numerare, our cryptocurrency, that actually had this massive impact on everything, where the people who stake their model with the cryptocurrency, basically they're taking a kind of bet on their model by betting cryptocurrency, those users end up having much, much better models.

And I think it was actually that piece of like game theory, changing the game a little bit, having skin in the game, that actually is the biggest thing we've done by far, because it captures kind of what's going on in normal hedge funds. If you invest in numerize hedge funds, I have a whole bunch of my own money in numerize hedge funds. So the people who invest are kind of investing alongside me, so we all have skin in the game and everything works. But if you don't have that, then things don't work very well. And blockchain is actually a kind of neat way to get people to have something to lose.

Can you describe that mechanic itself? So maybe say a little bit more about the cryptocurrency that you created, the genesis of that idea, and then this idea of staking. So I think some people listening will be fairly familiar with staking, but a lot won't. So maybe just describe that process and why blockchain and the cryptocurrency are uniquely set up to do that. Well, we started off Numeri paying people in December 2015. We started paying our users in Bitcoin. And it wasn't really like anything ideological or something. It was just like way easier to pay people in Bitcoin because our users were anonymous and they were all around the world. And so it was much easier to use Bitcoin for payments.

But because I had been involved and interested in Ethereum and invested in Ethereum, I was curious about how we could use Ethereum. And a lot of our early investors were people in the blockchain space like Olaf Carlson Wee, Fred Orson, Joe, crew these were all like my friends in Silicon Valley and so we started talking about could numera use its own cryptocurrency in some way and of course there isn't really any good reason to have your own cryptocurrency if all you're going to do is make another currency but if you make an application that needs smart contracts then having your own token makes a lot more sense and so with numera what you could do is stake your predictions that you send to us And if your predictions do well, then you earn more money.

And if they do badly, then we destroy your stake. So you'd never want to submit a prediction unless they were good, because you wouldn't stake it unless they were good. So it's a big filter for us. And the problem with something like Numeri or any crowdsourcing on the internet is that the much of the cost of spam for the spammer is low. It's easy to send millions of emails. I'd hope that someone clicks on one of them. it's easy to make loads of accounts on Twitter. All of these kinds of things are kind of what we're used to with the internet.

But in the real world, because there's a cost to doing things, people do the right thing. And that's what we wanted to make happen on Numeri. We wanted to be a little cost. Okay, are you willing to put $10 on your predictions? Because if you're not, we don't really want to use your predictions. And so that has this really nice impact. And why you kind of need a blockchain for this? Like you might say, well, what if you just wire you?

you the money, but somehow it doesn't work because what you're doing with a stake is you're having the blockchain be the escrow. Because if we just said, you know, wire money, we promise to destroy your money. What would that even mean? And we promised to give it back, well, how can, how can you know the promise is going to be honored? And if you do it in the smart contract, then you know everything's going to be honored. And therefore, the stakes on Numeri have been really large. I mean, over a million dollars have been staked on Numeri tournament. and a lot that's been returned and some of that's being destroyed. But it's amazing that you can do something like this now.

In terms of the efficacy, is there a bigger gap between staked and unstaked algorithms or between like small stake and large stake? Does the magnitude of the stake matter or is it more of a binary that just when people stake anything, you get much better results? That's a great question. I was very curious about how that would work. Anyone who's not staking, what we looked at, it was something like, and you get a sharp of one and a half. And then if you take the people who are staking, sharp goes to 2.1.

And that's a very big difference. You've got to think that the floor sharp is actually pretty high because the data set we're giving away is already decent, even if you just used regression. So you get like a sharp of 1. But then to go to a sharp of 2.1 is amazing. And that's just anyone who's staked more than one NMR. And what you find is if you say, well, what if they staked more than 50 or something, then you get fewer and fewer models. And those models actually don't work that well. So if you're staking a really large amount, it can often be something strange that you're trying. And then it doesn't work very well unless it's balanced out by others. So we actually see that it's good to use any model that's staked with any number of NMR. And if you

use them all together, then the average is really, really good. Do you still accept on staked submissions? Yeah, we do. We let anybody submit to us and there's no cost to making an account or submitting or downloading the data. And the reason you might do that is because you can still kind of get some practice. And if you win four weeks in a row with a free model, then you might want to like start staking and buy some NMR and start staking. So say a bit more about the full incentive structure here is for both staked and unstaked. So if I'm somebody that doesn't want to take anything and I just want to submit an algorithm and like you said I win four weeks in a row. Like is there a payout associated with that?

If you don't stake at all, you can win very small amounts for like verifying your phone number and then if you win like you'll get 0.1 NMR. And the problem with anything where you don't, when you're giving it out for free is that it's going to be like gained as much as possible. So we have to have those incentives be pretty low to prevent people from gaming it. But once a user starts to stake and starts to, then they end up becoming really, really active. That's some people who've never missed a stake since we started it. And the payouts there, what's the structure?

So do you basically win what you stake? How is the return on staking structured for stakers? Well, as with a lot of things on numerites, it's kind of complicated. Again, you have to be careful about people making multiple accounts. And so any system you design has to be like civil resistant. And I think if anybody's been following anything about Ethereum or Pitalik, what he's writing about, is always talking about can you make something civil attack resistant. That's because you basically can't rely on any identity.

So on Numerite, when you stake NMR, you're part of a staking auction. And in that auction, we're basically trying to allocate money to the people who are staking. And you kind of want the auction to be, there's no incentive. to make another account and submit again to kind of gain that. So that's, we do what's sort of a variant of a multi-unit Dutch auction, I think. Then people like do their stakes in that way. And in that way, like, the more you stake, the better. But it's also not like overpowering to really, really rich users who might stake loads just to try to win the whole prize pool. How is the price pool determined, like the size of it? How will that change through time or has it

change through time. And maybe just to put like some tangible numbers around this, what would be an example of a data scientist, I don't know if they're all anonymous, but some data scientists that's been active and how much they've taken home as a result of their work on the platform? Well, we definitely have actually had some that have made more than a million dollars, which is crazy. And that's because the value of our token has been very high compared to say how much it took initially to win some of the NMR before it was launched. Some users, did incredibly well. Now we pay about $5,000 per tournament per week, which is about $25,000 per week, $100,000 per month. So it's very profitable to people who get into it. It's quite strange to me

that when I started Numerize, very inspired by the Netflix Prize, which went on for a very long time and it was only a million dollars. And now Numerize paid out over $8 million since starting. And like you said, a lot of that, I know if you look at it, at the chart of Numerare, obviously it had like an insane, I don't know what the network value of it was at the peak, but something remarkably high. And as with all crypto, I think it's settled down lower than that. But it's this neat way of engaging a big and smart audience. Again, we've talked about the importance of the data there. I'd just love to hear your philosophy on what features and what targets to feed this system. I understand that a whole point here is that you're not going

to reveal like what the actual data sources are. But just like the philosophy around, you're, what to choose, where to look for it, how to source interesting data, what targets to tie it to. Like obviously, everyone you said before, maybe zero is bad returns and one is good returns or something simple like that. But there are all sorts of outcomes in markets that could be binary as your targets that you're trying to fit to. So I'd just love to hear your philosophy on how to choose the data, what to feed, and how to choose and feed targets as well. I think for any of this data, you want to make sure that it's normalized in some way. So I'll tell you, I mean, for Numerized data, one thing we, I think, have been open about publicly is that we don't use

alternative data or any data that's itself kind of already very new. We try to use very clean, very long structured data. And, you know, we're already doing a lot of complicated things in one company to then also use very, very complicated weird data. But once you have that, one of the things you're trying to do is make the past look a little bit like the present. So if you had a hundred pictures of people's faces, it would be way better if they were all looking in the same direction, had the same lighting. So it's way better if all the faces were like mugshots, because then they're all looking like they're from some kind of similar face distribution. But if they're people looking to the side, people with glasses on, people, then it starts to be irregular. And it's much better to have irregular data.

So you kind of want to make the past look like something like the present by making the past normal in some way and regular in some way. And that's what we're trying to do with the features. And the same thing for the target variable. It turns out you don't really want to make with hedge funds, you don't really want to make money when the stock goes up. You kind of want to do it on a risk-adjusted basis. So you want to make maybe your target of one should be something like a risk-adjusted return. and that's similar in kind of like your goal with regularizing your training data. You also want to regularize your targets.

So numerize just a very simple binary classification problem on lots of other financial applications. People actually use regression where, you know, you want to model the exact return. But we kind of like simplify it by making it zero or one. Am I interpreting the piece on data then the correct way when I think, okay, this sounds like if you've got a big, long history of stocks. There's a lot of data on stocks historically that you're looking at things like financial statements and price and things of that nature that are, I would say, let's say common in sort of the quant research world, things like the sources of data that led to the value factor and the momentum factor and things of that nature. Yeah, that is the kind of classic quant

style data where value and momentum and things like that are part of the literature of quant investing. So you sort of have to have those as a starting point, but you can do lots of other things as well. And again, none of the features on Numeri are that. Feature 1 isn't value or anything like that. But in principle, those are the very common features in the stock market. And so those ideas are somewhere in Numerize data. How much do you subscribe to this kind of old school wisdom of crowd model, where I think there are four or five components of required for a crowd to be superior to an individual.

It's like diversity of opinion. There can't be group think. They can't be overlapping. It's got to be decentralized. How do you think about that model? Is that something that you thought about when setting this up? Yeah. I mean, people often talk about the wisdom of crowds and they actually forget all of those requirements. You have to have certain things for it to make any sense. So if you poll people on the street, you say, what stock should I buy? And you ask 100 people, that's not going to be a good strategy because those people are very correlated with each other. And they all just say buy Tesla and sell Snapchat or something in this city. So you definitely need mathematically, if you look at what it means to average two things, if they are co-linear, the average is identical to just using one of them.

if they are orthogonal and they each have their own signal, then they actually going to be additive when you. So these things are like, yeah, very important in the way we think about this. How big is the group of stakers, let's say, how many people have staked any Numeri at any point? In the beginning of the year, we used to get 60 stakes per week, and now that's up to 600 stakes per week. So it's pretty cool that it's being used so much. And, It's way more used than most things on Ethereum. There's sometimes where there's more stakes in dollars and number than there are Cryptokitties being sold.

And so it's really interesting to see that maybe blockchain isn't about consumer things like Cryptokitties, but maybe it's about enabling weirder companies that couldn't exist like Numeri. It's definitely interesting that a weekly basis 600 data scientists would be submitting algorithms to a blind data set. It definitely feels like things like that will happen. in the future, like you said, maybe this isn't well-suited this structure to the healthcare world or other verticals. But it definitely shows just in raw numbers the power of the crowd. It begs the question for me

of as the person at the top of the centralized piece of this, which is, you know, taking these algorithms and actually turning it into a strategy for real money, how you think philosophically about the move from old quant, I'll call it, which was, you know, form a hypothesis, test it more linear, sometimes nonlinear, but more linear type, least square regression type work versus something now that is way less interpretable. So if you're working in 50 feature dimensions, we can all visualize that scatter plot, maybe a three-dimensional one. But once you get to 50 dimensions, in many cases, we have no idea really what's going

on in these models to come up with the predictions. And one interesting question in Quant is always, how do you know when it's broken? So if you put a model into production, how long do you give it to work? what are the signals that you look forward to unwind it and maybe move on to something else. I'd love to hear your philosophy on whether or not you care about interpretability of what the models are doing and how to know when something's broken. We definitely don't care about interpretability. And I think probably have taken that to almost as far as it could ever go.

If you think about it, the way Numerai works, like we do not build our own models. We just rely on the people to build the models. So the people who, the data scientists who are building on numerized data, they're operating on data that they can't see. They don't know what it is. And then when they submit predictions to us, they are not giving us their models. We don't know how they made their models. Oh, interesting.

They're just giving us the predictions. So we're like licensing predictions from them. But we never see their code. So what we are putting into production is the raw predictions without even knowing the algorithm that was used to create it. So we don't know what the model is that was built on data that the data scientists don't know either. So it couldn't be less interpretable. Do you get concerned at all from like a supplier power standpoint with that? It surprises me that you don't get the model itself. You're just getting the prediction. So how do you control for,

you know, the people that build the best model, not submitting their predictions anymore in the future? Did you think about having the process be that they actually have to give you their model. I'm curious about how you thought about that. Yeah, it does make a kind of tension, a good tension between us and our users because they really can have this genuine threat of not coming back if we don't pay them or we do something stupid. And I like that tension. And I think if you do, there are the companies sort of doing things similar to numerite, like there's one called contopian where they have all the user models just sitting on their service. And then they just say, well, we promise, you know, we won't use them without asking your permission.

But that's obviously not really how that's not really part of the crypto zeitgeist, which is like you actually want to be able to genuinely jump out of a system to express a certain thing and be engaged in a kind of trustless relationship. You know, we are not trusting our users with our data. We're totally obfuscating it. why should they trust us with their models? And I think that's the way things are going. And most systems that, you know, where you don't feel locked in, you end up being more engaged than if you were locked in in in certain sense. I remember that, I think with PayPal, there was this thing that the easier they made it to withdraw the money that you had in PayPal, the less people would withdraw it because they always knew there were all these ways of withdrawing. And I think that's a nice way to go

with something like this, make it as easy to leave as possible. You mentioned that you don't build models internally. Do you build a model of models on top of the ones that have been the best performing? Well, we actually did do a lot of complicated things with making a meta model that was based on doing machine learning on the user models to do the ensemble. But it turns out, yeah, the power of the NMR staking, it's very hard to beat, taking a simple average of the staked models. We have a few things that help, but that ends up being a really good baseline. And so, yeah, it's almost like the users are self-selecting by choosing to

stake or not. So they're telling us they should be in the meta model and what they're telling us is true. So looking forward, what is the most interesting, I guess, set of things that you're working on that you think might improve the overall process? So if the move from not having the cryptocurrency to having a stakeable cryptocurrency sounds like maybe the major, one of the major, if not the major evolution in the success of the model to a 2.1 sharp. What are the things looking forward that you think will keep you at the edge of the competition? Some big ones that we've done recently were improving the staking auction in some of the ways I described earlier where you make it rational only to submit your best model. Then that's had a huge impact, taking us even further if our sharp.

and then also releasing multiple tournaments. So right now, enumeride, there's actually five simultaneous tournaments at any time, and users can maybe be good at one, but not the other one. And so we're getting more diversity of models because the target variables are different in each of those. So that kind of thing is very interesting. We release the five tournaments, and in the first couple of weeks, it only took a couple weeks

for the average person to stake in 4.25 of the tournaments. So it's sort of like this amazing thing where our data scientists are totally ready for whatever data we give them. And they're willing to try lots of different techniques on all of these tournaments. So I think if you think about another hedge fund, imagine you had a hedge fund that had a very good global equity model. But then they decided, you know, we should do a currency thing. that would sort of take many years of hiring people and getting a whole bunch of new things set up

before they could even do anything. And for us, it's like, oh, we found a currency data set. Let's stick it on and see what people can do with it. And then very quickly have the best model in the world for that data sets. So it's really nice to be able to do that. So there's lots of progress we can make in that dimension by just releasing new datasets. And every single thing we've done like that has improved. the shot. So it's not clear where that will end. And then going forward, I think the one interesting

thing that we've sort of hinted at, which we haven't announced yet, is the idea of, could we crowdsource the data part? So we are crowdsourcing models. Could we get people to give us data? And if you just did that simply like, hey, hey, everybody who has a data set, email me, and you have the same problem of you'll get a whole bunch of bad data, a bunch of spam. And maybe you can use blockchain, maybe you can use NMR to give predictions of different kind. And so that's something that's super interesting and something we've been thinking about for a very long time. Is that sort of like an Oracle problem effectively in crypto? It is in a certain way, yeah.

We would need to trust something. But I think you can basically, I think the, do you know George Akeloff? Yes. The name sounds extremely familiar. Market for lemons. He's a Nobel Prize winning economist. He had this thing about if you have a market where there's lots of bad cars, then none of the good cars will ever be in that market. And that's in financial data, there's a similar thing happening where if someone were to email you and say,

hey, I have an amazing model that predicts the stock market perfectly. You know, you don't even reply to that email. You don't even bother. Even though what he's promising has insane value, the probability that he's actually got something is so, so low. It's not even worth considering. But maybe if he emailed you and said, I have this really good model. and I've actually staked 10,000 NMR on the fact that it's good. Maybe you'd actually want to read that email and spend a little bit of time looking at his data

considering you're playing a different game now. You're playing a game where he's actually got something at stake. And I think that's pretty cool. And I think you can extend that to basically where there's market collapse. There's huge markets do not exist for, I have signals, stocks. It's always so many questions. Are the signals really going to work? Does the person, why isn't the person just trading it themselves? All these things that make it impossible for good markets to form around that. I think those markets would form if you had stakes attached

to all of them. Really interesting idea. And so I want to ask a clarifying question. So the way you're describing it sounds like kind of, because you mentioned earlier, don't use alternative data sources, maybe stuff that hasn't been around all that long. But this sounds like you might allow someone to come with a predictive signal, but that's based on their own data, so long as they staked it? Or is it just that they have some interesting data and they're staking the idea or they're willing to put a stake behind that if they feed a raw data set into the other part of your engine, the data scientist, that they'll find something compelling. Which of those two is it?

It's really more just if right now on the website, when you upload predictions, you're all these abstract security IDs, you know, like you don't know what they are. But if you could upload to Numerize websites that you think Apple's going to go up and you think Google's going to go up and you can actually give real ticker symbols, then you could be generating that up and down signal from arbitrary data that you have, of data whatever. So it's both. At the end of the day, the submission is still a signal, not just a raw data set. So someone couldn't say, you know, I've created this amazing, you know, whatever it is, data set on all companies, but I don't know what the hell to do with it. So, like, have your data scientists figure it out. Instead, it's, no, just do

whatever you want to do, but you're submitting a signal and you're staking something on the predictions embedded in that signal. Yeah. Yeah. Understood. I think, like, opening Numeri up to let other people use it is so, like, well, to let other companies use it or other industries, I think it's just so the wrong approach. Instead, it's more about individuals. It's like if you're an individual and you have some data that you want to follow the protocol to upload it, then this could be a really good place to put it, especially if you're willing to stake on it. I would love to hear your impression or take on the rest of the maybe even specifically quantitative investment community. You said you came from a sounds like a more traditional hedge fund where you were working on machine learning when maybe at the time not many other people were.

You mentioned Quantopian, you know, 2 Sigma. You've mentioned some of the names in the quant world. I'd just be curious to get your take on where you think that world is in relation to some of the topics that we've discussed. So how many of the big players are actively using machine learning techniques versus the kind of old traditional way of thinking about quant? Yeah. I think there's definitely machine learning is not as well used as people might think by the big funds. They might have teams that are sort of doing machine learning, but probably none of their stuff's actually in production. And I think places like Renaissance don't use machine learning from what I know, from some of the people who work there.

That's just not what they do. And one of the reasons for that is that they just kind of have an edge. If they have a really good data edge, and suppose they've been just collecting stock market data, long before anyone else and actually storing it in databases long before everyone else. Their data edge is so big that they don't need their modeling edge to be very big. That's sort of counterintuitive, like I guess the brand of Renaissance. It's not like, you know, we have the best data. That's not what they're saying.

They're just like, we're the smartest people. But maybe they just have the best data. It's why I asked the question earlier about like if you had a pie chart of the importance of the data you have versus the, I guess it's the informational versus analytical or modeling edge that everyone always talks about, which is more important. It's, you know, it's interesting that maybe places with all these PhDs, maybe it's a data advantage, just not a modeling one. Really interesting.

What else is exciting to you in the, I guess, the hedge fund world? It's interesting to hear all of the, these very crypto-like philosophies applied to a very old-school, unchanging model. So are you encouraged by that overlap? Are you frustrated by it in any interesting ways? I'm assuming you're set up with prime brokers just like everybody else and you're in the same, you have the same legal teams as every other, you know, headliner hedge fund. I'm just curious your take on the intersection of those two worlds and any surprising reactions you might have had to them. I think one of the interesting things about the crypto

when you're designing something in crypto you're really thinking about what's the best thing you're not thinking about making money and this couldn't be more true of say someone like Vitalik who when you're thinking about designing proof of stake

or something like that it's like the last thing on your mind is like money it's just like how could you get the system to work and then all these different actors to behave in a certain helpful way And I don't think anybody has thought about the hedge fund industry with that lens. So I'm trying to be the vitalic of hedge funds and basically look at the stock market and say, well, if you're an alien coming to Earth and you're like, well, clearly capital allocation is a super important problem.

Capital, you need all the money to go to the right places. Otherwise, society won't move forward and technologies won't be created. And so this is like a really important thing. And the way that the humans have set this up is, well, let's have a huge weird zero-sum-esque game where there's just like all these smart people from the top universities joining hedge funds that are identical to other hedge funds in every single way except that they're friends at one and they're at this one. And they're all buying the same data and none of them are sharing any knowledge about what's going on. It seems like a shelling point. It doesn't seem like a supernash equilibrium. There's probably some technologies we could be used to make all this much, much better.

Can you describe a shelling point, just not to take it for granted? Something that is an equilibrium. I don't have the perfect definition in front of me, but something like it is an equilibrium, but is kind of like a local... Local optimum or something. Like it could be way better if certain things changed. And that's what Numerite has been doing where you have things like, well, we want you to help work on this data with us, but we don't want you to write.

run off with the data. And if we could just solve that, we could have a completely different system. We actually have a distributed hedge fund, which is never being done. And so that's the problem we first worked on. And then we're like, well, now how do we get people to really believe, express their belief? And then we're like, well, what if we used numerare and made a cryptocurrency that did that? And that, yeah, so the kind of game design of the company is the company. We're not trying to do anything except make a system that is resilient and long term going to be way, way better. So yeah, that's sort of my take on the hedge fund industry. And I like that in a way no one else's is coming at it from that perspective. One kind of final question on your strategy

is, and I'm obsessed with marketplace businesses in general. I just think it's so interesting when a business has to build two sides of a market effectively, you know, yours being data scientist on one side, sort of investors on the other side, and you're the thing in between, that the concern would be somehow a competitor comes up and spins up something that pays out more or in some way as caters better to the data scientists as a better audience building strategy. So I'd just love to hear your thoughts on that, like how you reach data scientists, how they become aware of numerai that's very intentional versus just word of mouth. Maybe it is just word of mouth, but if it's not word of mouth, like strategies you have for

building that audience and for keeping them happy given that they represent such, I think, a large portion of your advantage. We didn't do any advertising or anything like that, but Numeri, because it had all these unique properties, ended up... It's like a PR dream. Yeah. Yeah, it sort of became like there was a lot of press about Numeri. And it's very interesting to me how you can have, if you ask a Silicon Valley software engineer,

name three hedge funds, that Numeri would maybe be one of them. It's kind of crazy. But yeah, that was a big part of getting users. And, yeah, I mean, we could, we've been doing, nearly everything we've been doing has been like zero to one. So we have made some mistakes in some of our users get grumpy with us about certain things. And we have been kind of innovating faster than we've been maintaining in some sense and in some parts of what we're doing.

But I do think that our users are, you know, at any given time, if you're holding NMR, it's because you think it's better than selling it. And we have loads of users holding NMR and staking it every week, and it's up 10X since the beginning of this year. So I like where we are in terms of all the important things. And, you know, I've never understood cryptocurrency valuations. So it's hard to know, hard to think clearly about that when they're so correlated, even though numerize so different.

But, yeah, I think it'll be hard for another company to come and do something similar to Numeri. I think there's a lot of things under the hood as well that can't be seen. Well, definitely an interesting intersection of some of the more important technologies of our day. So it's been interesting to hear about everything you're doing. My closing question for all my guests is to ask what the kindest thing that anyone's ever done for you is. Coming to mine is like the early parts of Numeri where I came to Silicon Valley without any connections here. and I was coming from South Africa. And Howard Morgan, the co-founder of Renaissance Technologies, was our first investor.

And he kind of has a similar thing, I think, where he's just not, he's not in a time in his career where he wants to just invest in things that are nothing but making money. He wants to invest in very interesting things and things that could be really interesting for the world and society and things like that. So he was extremely kind to me by investing. and numerai.

Wonderful. Well, thanks again, Richard, for your time and hope to catch up as you guys continue to evolve. Yeah, thank you. Hey, everyone. Patrick here again. To find more episodes of InvestorFieldguide.com forward slash podcast. If you're a book lover, you can also sign up for my book club at investorfieldguide.com forward slash book club.

After you sign up, we'll receive a full investor curriculum right away and then three to four suggestions of new books every month. You can also follow me on Twitter at Patrick underscore Oshag, OSHAG. If you enjoy the show, please leave a quick review for us on iTunes, which will help more people discover Invest Like the Best. Thanks so much for listening.