Effectively Wild: A FanGraphs Baseball Podcast - Effectively Wild Episode 538: Nate Silver, By Popular Demand

Episode Date: September 19, 2014

Ben and Sam talk to Nate Silver about baseball analysis and the first six months of FiveThirtyEight.

And the whole world slept that perfect night. I learned a little about living and a lot about life. Out there in the backyard, just me and him and 538 stars. Good morning and welcome to episode 538 of Effectively Wild, the daily podcast from Baseball Perspectives, presented by The Play Index at Baseball Perspectives, presented by the Play Index at BaseballReference.com. I am Ben Lindberg of Grantland.com, joined as always by Sam Miller of Baseball Perspectives. We have got a guest for weeks, I want to say months, some of the backseat bookers in our audience have been looking ahead to this episode and tweeting and emailing and Facebook commenting

to tell us that we should get Nate for episode 538. It's a little gimmicky, a little bit on the nose, but we are happy to have the excuse to talk to someone whose work we admire. So we are pleased to be joined by the former managing partner of Baseball Perspectives, which is probably the first line on his resume, still right above projector of presidential elections and author of The Signal and The Noise and founder and fearless leader of FiveThirtyEight, Nate Silver. Hi, Nate. Hey, how are you guys?

Great. So I believe if Wikipedia's information is correct, that today is the six month anniversary of the launch of FiveThirtyEight's ESPN incarnation. So if I'm right on that, congratulations. Thanks. My managing editor, Mike Wilson, pointed it out to the staff today. It's one of those things where it doesn't feel like six months. It feels like some days like six years and some days like six days, basically.

But it's been a lot of fun. I hope that people see that we're, you know, have a slightly larger repertoire in terms of what we can do now and get kind of a thesis to the site, but I appreciate it. So that's kind of what I wanted to ask you because we talk a lot on the show and sabermetric leaning people talk a lot about when certain statistics stabilize, how big a sample you need of swing rate or strikeout rate or another stat before you can be confident that performance over that span is representative of his true

talent. So what do you think is the period that it takes a website to stabilize? How long does it take before a website's content or audience is representative of the vision that you had when you set out? So Bill Simmons told me before I actually decided to go with ESPN that he thought for Grantland it took, I'm not trying to get any confidence here, but for about 12 to 18 months for them to kind of feel like they fully hit their stride and then still continuous improvements on top of that. It's only about three years old itself. I think I was kind of naive about like, I think I said in some interview that,

oh, we're like 75% or 80% ready. I think I said before we launched. And what I meant by that was that we had hired like 75% of the staff that we had budgeted for. But you don't really know very much about how to run a website or produce any type of product until you actually have customers. And when you're kind of drawing a lot of things on a whiteboard and doing kind of all these J-school critiques,

you're kind of dumb until you actually see what works and what doesn't. One thing we found, for example, is that readers really like when we go and actually do some, you know, quote-unquote traditional reporting, which is something BP has done more and more of, too, over the years. We sent, for example, a reporter to Ferguson, certainly not to report the front line of that story, but to talk about the economic context there and about how this had been actually a center for protests over minimum wages in the fast food industry. And it's

a symbol of how a lot of suburban areas are working. So stories like that, where it's done by a numerate reporter, but it's still traditional reporting, are something people like quite a bit. So when you're assessing where you've come in six months, do you sort of see the topics that you guys approach getting narrower as you find your real strengths and the topics that really get a particular response from your readers? Or do you see it going exactly the opposite and getting broader and broader as you try to kind of eventually take on the whole world? Yeah, the goal is to be pretty broad. You know, we're kind of roughly broken down, I'd say, a third sports, a third politics and economics, and a third kind of lifestyle and science stories that can be highbrow or

lowbrow or whatever else. But, you know, the idea, I think, originally, I mean, look, if you look at the kind of stuff I wrote at BP back in the day or a lot of stuff there now, a tremendous number of pieces are things that are like, okay, here's some original data that I pulled, and here's a regression analysis, and here's a punchy conclusion about it or a cautious conclusion. And I totally love that stuff.

That stuff is great. But we want to expand it up to where it's not the only kind of thing that we do. So sometimes it does involve more traditional reporting, also reporting on things that are just of interest to geeks when we post stories on Scrabble, for example, or burritos or how to find the best airfare, things like that. People tend to gravitate toward those too. We're launching a film series, which will be modeled after 30 for 30. The first movie comes out, I think, next month. So finding different ways to reach people and also kind of, you know, figuring what are the things that you don't want to do. We don't do a lot of aggregation. I mean, I don't have like a huge moral problem with it or anything, but we want to be a place

where we're kind of doing more of the legwork ourselves and either taking an original research finding or doing original reporting. We definitely shy away from fund-a-piece stuff. And believe me, there are a lot of times when I would love, if something happens and you don't have the time to be that thorough, I'd love to just kind of spit out my first thoughts in 500 words, but that's kind of not what we're about. So, you know, we try not to absolutely prohibit everything. You don't want to let the philosophy of the site get in the way of itself. But, you know, mostly we feel like what we do is in some sense kind of hyper-traditional,

that we believe in doing original work ourselves. We believe that kind of the world's a complicated place and you want to trust the reader to be able to untangle things for themselves. It doesn't mean you're just kind of presenting your notes in an unedited way. But there are a lot of times when we write articles that are saying, hey, actually this thing is a little bit more complicated than people are saying. And, you know, that's a little different, I guess, than maybe some trends now where it's kind of like, oh, here's just the kind of summary, the bullet-pointed version. You know, we want to kind of lower the fourth wall between us and our readers and trust that our readers are really smart and have more detail about it than less.

And sometimes it's in the footnotes or in offshoots to things that we do, but a big kind of premise around here is show your work. I think trust in journalism ultimately should come from verifiability, and that can be by being clear about your methods and your sources. It can come sometimes by making predictions. That's one type of verification. Sometimes by publishing data or code. We do that not with every story, certainly, but fairly often on GitHub and stuff.

So people can kind of see not just what you concluded, but walk with you on why you're there and, you know, frankly, what the problems might be with your approach. So when the subject matter is as diversified as it is, and there are five main subject areas on the homepage, but one of them is life, which encompasses everything. So do you count on a core audience coming back every day to the degree that you would say at BP where there are subscribers who will show up every day to the degree that you would say at BP where there are subscribers who will show up every day because they know that there will be baseball content or maybe at the old 538 where there was a higher percentage of politics coverage people would come for that it was the destination for that do you get more hop-ons now to borrow an arrested

development term or are there still kind of the core people who come to 538 every day just knowing that there will be something there that appeals to them so you know we get um we get stats from different platforms on it kind of breaks it down into new readers recurring readers and loyal readers and i on an average day it's like 40, 40, 30 between those categories. I don't know what everyone else's traffic is. I think that's on the side of having somewhat more loyal readers, probably not quite as much as BT has or something.

But what we'd like to do is make sure that when people click on the site at Bandim, they'll see a couple of things that will interest them. And obviously, we go very deep into some subjects like the midterm elections and the World Cup and so forth. But we're also of the view that everyone is kind of a grazer now in terms of how they consume content. And people spend maybe a a couple minutes on every site from BuzzFeed to the New York Times, right?

And so we're not under the illusion, I hope, that we're the only place people are looking for content. And some of that means if we don't think we can cover something well, then we, you know, maybe sort of take a pass on it. Like we would love to have Interspec done more with the Scottish referendum, for example. But it's a hard thing to model. There's not a lot of history you can compare it to.

And we didn't want to put something together that was kind of half-assed if we couldn't cover it in the right way. So we had a couple of contextual stories, but those decisions matter a lot. But overall traffic is really substantially, like an order of magnitude or so higher than it was when you're just a politics site. So I hope that we're kind of keeping the politics readers and also kind of having people who like the philosophy of the site, or just if you read one article at 538 and you enjoy it and you never come

back again, I'm still happy to have you as a customer, so to speak. So your baseball analysis, if I can turn it to your baseball analysis for a minute, your baseball analysis is really from a particular generation where there was a lot of interest, a sort of huge, almost exponential growth in both awareness and information in the sport. But just before the kind of super micro pitch effect stuff took over and it became, you know, a big industry of heat maps and such. So do you feel like there is a kind of best era to have been a sabermetric writer of, you know, between the 80s, the 90s, maybe the early mid-2000s, now and in the future. Is there one that you think

is kind of the richest vein for research? I mean, I think now is my default instinctive answer, right? You know, when I kind of moved on from BP in like 2008, it wasn't quite a point of stagnation, but it kind of felt like, you know, the box score stats had been exhausted, there were some good prediction systems out there based on those, there were

Boris McCracken's findings, and you were starting to have next gen stats, but there wasn't kind of enough history to really do cool things with them. But I'm always, like, super impressed whenever I read just how advanced this stuff is with respect to measuring catcher framing or defense or, you know, approaches to clubhouse chemistry or whatnot, right? I mean, it seems really interesting to me.

It also makes it competitive. We probably have had, in some ways, we have Jonah Carrington and stuff for us, Neil Payne and Ben Morris and freelancers, but in some ways baseball is a little tougher because competition is really good, right? Whereas in the NFL or the NBA to a lesser extent, you

can do something pretty good that kind of works in a daily news cycle and it's still somewhat novel, but boy, the bar is really high in baseball, I have to say. Does your brain still generate research ideas that maybe you'll never follow up on or maybe that you'll poke around on the side but never publish? Do you have a desk drawer filled with half-done studies that are just kind of a hobby now? Yeah, definitely, whether it's sports or elections or whatever else. Because for me, the fun part is the moment of discovery. When you ask a question

and say, hey, let me go and grab some data and test this. And then sometimes it becomes tedious to write up the finding if it's no longer exciting for you. So, yeah, there's lots and lots of spreadsheets or half-written memos that have never turned in to anything on the site. But I guess it's kind of a hazard of the job. I mean, this is basically all about kind of asking good questions and being curious about the world. And so, yeah, a symptom of that is that um sometimes there are more ideas both

for me and for the rest of the group than we have time to really execute on every day is there is there any sort of falsehood that you see repeated over and over in you know baseball writing right now where you have a spreadsheet that would contradict it and you sort of think oh there they go again getting it all wrong. Only I know the truth. I mean, I think there is maybe relatively more room for improvement on covering sports finances. Like one thing I found, for example, is that when a team either signs a marquee free agent or loses one, it seems to have some effect on ticket sales, controlling for a one-loss record and making the playoffs and so forth.

So, you know, kind of that paradigm gets a little complicated where you're thinking about, oh, do we have a more marketable player? For example, if the Red Sox re-sign to a 35-year-old veteran who's popular to a contact at war says it's a terrible idea, but he still gets people in the ballpark and stuff like that. It's maybe that, you know, how do you approach those types of questions? So the sports finance stuff is something we try and do more of. But there's not a whole ton of really super exciting findings, I guess, on the baseball side.

Maybe more on the election side. I've been sitting on this thing for a long time. It talks about which party really has an advantage in the electoral college and how does that rate relate to turnout. And it looks to me like the fact that Obama swept all these swing states in 2012 had more to do with his turnout advantage than anything inherent in the electoralctoral College itself. So that's an interesting thing that might take 4,000 words to write about. I'll probably do it at some point. But part of it, too, is that because baseball season and election season kind of coincide,

it's, you know, frankly, I probably watch more basketball and football than I do baseball right now because then it's kind of a slow period for elections and baseball. Like, you know, the climax of the season is right when I basically am sleeping two hours a night. What is your perception of the broadcast bubble, if that's what you see it as? Teams getting incredibly wealthy on these massive local deals and national deals. Bubbles are something that you wrote about in your book. So I'm curious if you see this going the same way. Obviously, it has broader implications than just baseball, but it has become a major issue in baseball specifically. Yeah, to me, I mean, anytime you see the price of any commodity increase

really fast, a lot faster than its baseline rate, then you should be suspicious. It doesn't mean it's always a bubble, but the probability is fairly high, I suppose. I think one thing driving this, too, is that the rise in value, which has been rapid in sports franchises, seems to track really well with the rise of the global 0.001%.

Because if you literally now have to be a billionaire to buy a sports franchise, then there's much more of a market for it when you have more billionaires. It's almost like the high-end art market or something, where only a certain type of people have access to it. And that's part of what makes it valuable, is that you can kind of pass on this commodity to other very wealthy people and make a bet on what long-term trends in wealth look like. But I would think there might be a correction at some point in franchise value.

I mean, the Clippers sold for what was it, $2 billion when Forbes, which the rest of them had their problems, that it was maybe half that or something for a profoundly troubled franchise. So yeah, I think there could be a correction there. I get worried whenever you see people who are making financial predictions, and we're going to grow it at inflation plus 3% continuously. I don't know if that's necessarily going to be the case. I want to ask you about war, because every person who has a byline is going to be writing something this year about what they think about war. And it seems like the general feeling in most of these columns is that war is probably maybe the best stat that you can get, the

most all-inclusive stat. There's nothing exactly wrong about it, but that there's too much confidence in it, that it's leaned on too heavily, that there are clashes in the data that make it unreliable and creates only the illusion of reliability. Do you have a sort of feeling about the way that war has developed into the number one stat in the last five, ten years, and how it's applied and how it's used? I mean, it's not like war is just something people pick out of a grab bag. It is supposed to be the summary stat to the extent you can have one.

I mean, a lot of this comes down to where do you need precision? And maybe you can't have a certain amount of precision, right? But if, you know, if you write a piece saying, oh, this guy was half a win better than this other guy and then come to some really profound conclusion based on that, then you probably have to be pretty careful about that. But as a general guide, I think it does totally fine. You know, I think to have different versions of it out there is actually helpful.

And by the way, the ones in baseball, I mean, the differences aren't nearly as profound as in the NBA, where depending on what system you use, what type of data it uses, you have really profound differences of opinion about whether players are even above average or really poor or really good. So baseball, I think, is fortunate in that sense as compared to football, as compared to hockey. I hope hockey will get better data, but the stuff they're doing is super primitive for the most part right now. So there's a little bit of kind of angel dancing on pinhead debates about war.

It's a pretty good system. This is about the time of year when Sam and I often start talking about projecting playoff success and the difficulty or impossibility of doing that. And I know that Jonah and Neil wrote about that yesterday, and that was interesting. And you, of course, created the secret sauce, which was subsequently discontinued at BP. And I don't know how you felt about that, whether you felt that that was premature or not. But do you think there is still something in that approach of trying to do a regression that will give you a secret to playoff success? Or is it not really a pursuit that we should be going after anymore?

really a pursuit that we should be going after anymore. You know, as I've kind of evolved and done more things in different ways, I tend to be more suspicious of kind of data mining exercises of that type, where like, oh, let's go test out 30 things and see which ones have some significance value, because you're going to have, you know, if you test out 50 variables, then two or three of them are going to come up positive just as a matter of chance, right? So you have to be careful about that. I thought the insightful thing in kind of the book chapter I wrote about that is that, you know, things are slightly different when you're playing in the

playoffs and that, number one, you're playing against other very good teams and there's some evidence that when you're playing against other very good teams in the there's some evidence that when you're playing for the very good teams in the regular season also that it tilts things slightly to pitching and defense. Obviously, you have a more top-heavy pitching staff in the playoffs, and that can matter some. Players are better rested, generally speaking, but also fatigued in other cases.

So I think that's worth talking about. But a lot of these things are, yeah, it might make a 1% difference here and there, but there's probably not any secret sauce per se. The sort of conventional wisdom or I guess the lore around baseball is that if you have an ace or maybe even a couple of aces, that that makes you extra dangerous in the postseason. And it has kind of been an ongoing mystery of this podcast that ben and i can't quite figure out why there doesn't actually seem to be any correlation between the strength of your ace and how well you do in the

postseason it's not actually a a particularly beneficial variable and we've had many different hypotheses come up do you have a hypothesis for why two teams that are essentially equal but one has a super ace that can pitch a disproportionate number of innings in the postseason compared to the regular season doesn't actually have a better chance than the team with the lesser ace? in the data too. You would think that as pitching staffs get compressed that it's kind of inevitable that a team that can start its number one starter three times in a series or twice at least is going to benefit from that to some degree. But yeah, I don't know.

I guess it would be suspicious if you didn't find that if you had a large sample of data. But who knows? It probably depends on how you define number one starters to... One theory I have, by the way, at least curiosity I have that I haven't done enough work on myself, is how much of the resurgence in pitching has to do with the fact that

we now understand pitching stats a lot better. Maybe you guys know the research on it. I'm sure you do. But, you know, if teams are going by fifth, basically, instead of ERA or wins, and they kind of make those decisions correctly, better predict who's going to be good, how much of a league-wide improvement in the ERA would that lead to? Maybe not a ton, but probably something at the margin. And also, people figuring out, I think, how to use their relief pitchers really effectively, maybe which guys should be relievers and starters.

That seems to play some role. Do you feel, you mentioned being aware of the research and how hard it can be to keep track of that now that there are so many outlets doing this sort of work. Do you feel that when FiveThirtyEight does a study or when some other major media site does a study, is there a responsibility to sort of scour the internet to see whether anyone has ever done any work on this subject and whether it's been done before so that you can cite

and link to that work? Or is there value to just sort of doing it originally and redoing it and maybe doing it in a different way and bringing it to a bigger audience and not necessarily exhausting all the resources that you could to find out whether something has been done in that area already? It's a great question, and there are kind of two parts to it, one of which is kind of maybe a journalistic ethics thing and one of which is a research question. In general, it's good when you have people approaching the same question in largely the same way or somewhat the same way, but you have better replicability if you do that, and that's a good thing.

People have found that in academic context, replication rates are quite low. If people try and recreate the same study in medicine, for example, it fails to find a positive result a lot of the time. Sports data tends to be a little bit more robust than that. But that's an important thing to do. It's also the question of, you know, we've written things and the people write us later and said, boy, I did something pretty similar here in 2011 and you didn't cite me, right? And we're like, well, if the internet's a big

place, you know, we're sorry about that. We'll try and drop in a link if we can. And about as often, we've seen someone say, hey, boy, you know, this is a really new and novel analysis that we've done and isn't this cool? And we said, actually, we did that in 2011. Why didn't you cite us, right? And that's super annoying to be on the receiving end of that. But I guess I'd say, you know, I don't have any commandments that you have to do a literature search before you start writing about something. If you are aware of something and you don't mention it, then I think that's a little bit misleading. But but to some extent of like i don't know i'd like to have a topic uh... myself and and uh... if you have a really surprising finding that you might

go more and say hey what's the previous research look like on their and if it's different but that might be something wrong they did something wrong the data is just really going to your or whatever else so sort of along those lines uh... last december Andrew Koo wrote a piece for us at Baseball Prospectus about the A's and how they had seemed to have found sort of an inefficiency in signing fly ball hitters who had a kind of a persistent platoon advantage against

ground ball pitchers. And this seemed to be like a real valuable thing that they had discovered. And everybody celebrated this article. It was interesting. It was a great piece. And a few weeks later, randomly, I was reading through your old pieces, and I found in 2003 that you had kind of written in a theoretical way almost this exact same thing.

You and Gary Huckabay had written a piece that was exactly about the platoon advantage that fly ball hitters have over ground ball pitchers and uh it kind of blew my mind and it's what really blew my mind though is that for 10 years or almost 10 years the idea had gotten really no traction it had remained kind of underutilized not talked about do you do you feel like there are like a in your heart you feel like there are a bunch of pieces that you wrote long ago that never quite got traction and would be just as timely and just as interesting today and maybe just as useful for a team

or for a writer to know about today as back then? Well, I think about Bill James. There's a lot longer than any of us in how contemporary this stuff still feels, if you read stuff from the 1980s or really almost anything that he's done. Yeah, people kind of only hold so many ideas in their head at one time, and kind of executing on strategies is a lot different than researching them. So it wouldn't surprise me if there are a lot of kind of nuggets buried in different things.

You know, I know with like building independent pitching that, you know, I guess you could probably find some predecessors to that if you dug back far enough. But it's also a matter of being skilled, I guess, at presenting things and thinking, hey, this idea is actually kind of a big deal and here's what the impact might be. So it wouldn't surprise me at all if people just said, oh, we're not going to do any new work for a year. We should go and reread things that we did before. Oh, yeah, you could probably do almost as well in the short run.

And then the research well would dry up and you'd have to go find some new things. So lastly, I've heard legendary tales about the last time that you ran Pakoda, which I believe was after you had already been pulled in many other directions. And I was an intern at the time, I think, and I remember hearing about epic three-day straight staying up playing Guitar Hero

and eating Chinese food to get the projections out by the time the book was due. It doesn't sound like the sort of thing that you would ever want to return to. But if you were ever to return to baseball projecting in some form, are there data sources that have become available since you FBP, since you handed over Pocota that you would now be interested in incorporating into the system? you handed over, Pakoda, that you would now be interested in incorporating into the system? Are there analytical techniques that you have picked up in your other work in other fields that you would try to apply to baseball?

What would a 2014 Nate Silver version of Pakoda look like? Yeah, I mean, I think I would definitely want to make a lot of use of next generation stats. And to be honest, I'm not completely sure about what everyone else is doing. I know like, you know, I think Steamer, for example, maybe uses data about fastball velocity and so forth that I think would be very useful. But yeah, one of the things about Pocota was that it was all built based on historical comparisons, going back to World War II.

So it's kind of like if any stats didn't exist as of 1946, it didn't really use them. And I think now that's probably the wrong course of action, that there's so much information contained in this new data that we didn't have before. And now we have enough years where it has been collected where you can go back and do some modeling and backtesting. Backtesting is not always a perfect thing, but you'd rather do it than just guess at how important things are.

So, yeah, it'd probably be built fairly profoundly from scratch, I would think. The other thing, too, is that in Dakota, you know, everything was kind of a giant Excel program and, like, it was kind of loose bits of static code that I had to recreate every year. So just kind of realizing that, you know, making an upfront investment in terms of coding things properly, building an actual program instead of doing things haphazardly saves a lot of time, a lot of time over the long run.

And in a given year, you might say, well, I'm going to hack this together and it will save me two days. That means kind of two weeks of misery every year when you're recreating your steps from before. And also, you know, if you're doing a lot of things by hand, then there can be errors introduced in different ways. And Dakota was kind of like one of those serial Christmas tree light strings where if you're like, oh, shit, I went and mistalked the park factors. Well, it's one of the first steps and you have to go back and redo everything. If I actually had a proper program like I do now for our election stuff, then that fix can be made in five minutes.

All right. Well, thank you for coming on and making a podcast return to Baseball Perspectives and doing our gimmicky branding 538 tie-in episode. It was good talking to you. Awesome. Thank you, guys.

All right. That is it for this week. You can find us on Facebook at facebook.com slash groups slash effectively wild. You can send us emails for next week's listener email show at podcast at baseball

prospectus dot com. You can rate and review and subscribe to the podcast on iTunes and you can support our sponsor by going to baseball reference dot com subscribing to the play index using the coupon code BP and get the discounted price of thirty dollars on a one year subscription. Thanks for listening. Have a wonderful weekend. We will be back on Monday.