Dan Snow's History Hit - Big Data and History

Episode Date: December 9, 2020

Dan Hoyer and Peter Turchin joined me on the podcast to talk about the new transdisciplinary field of Cliodynamics, which uses the tools of complexity science and cultural evolution to study the dynamics of historical empires and modern nation-states.Subscribe to History Hit and you'll get access to hundreds of history documentaries, as well as every single episode of this podcast from the beginning (400 extra episodes). We're running live podcasts on Zoom, we've got weekly quizzes where you can win prizes, and exclusive subscriber only articles. It's the ultimate history package. Just go to historyhit.tv to subscribe. Use code 'pod1' at checkout for your first month free and the following month for just £/€/$1.

Hi everyone, welcome to Dan Snow's History Hit. We've got a new way of looking at history on the podcast today. A new transdisciplinary field has emerged called Cleodynamics. Imagine just being a founder of a transdisciplinary field, let alone calling it something as cool as Cleodynamics. Cleo, like it, good word. It reminds me of the old Athenian general, not a big fan of his, but it's a strong, it connects a strong one. And then Dynamics can't go wrong.

Every sci-fi story has got a company with Dynamic in the name of the evil corporation. So Clio Dynamics, strong. And Professor Peter Turchin is the founder of that new transdisciplinary field. He's published lots of books. And Dr. Dan Hoyer works with him. The two of them are harvesting data. Of course, what else is anyone doing in the world? We're harvesting data.

Big data about the past. Every single piece of data they can find, they're scraping. And they're seeing if they can then tell new stories about the civilizations of the past. Big data has transformed every other field. So can it transform history? Well, see if you're convinced, have a listen to this podcast in which I interview both of them. Excited that many of you fed back positively about the live tour. It's going to be fun. Next autumn, next fall. It's sadly just the

UK at the moment, but we'll see. We're going to be going to lots of big cities. We're going to be talking to wonderful historians there, recording those interviews for the podcast, watching some video, going on little adventures in each of those cities and then learning about the history of the cities themselves. So please go and check that out. Tickets on sale Friday. It's going to be fun. And if you want to top up your Christmas present giving, go to the History Hit Shop. You can buy your comedy knitted knight's helmet or your gigantic poster of the Roman Empire at its peak. I mean, as I'm saying those things, it just seems too good to be true. What more do you need?

As the high street collapses, history it steps up. In the meantime, everyone, enjoy this interview with Prof Turchin and Dr. Dan Hoyer. Dan and Peter, thank you very much for coming on the podcast. Yeah, thanks so much for having us. Glad to be here. Now, you guys, what are you doing with your statistics and your data? Because I've been reading history books for a while, and everyone has a few tables in them, a few maps.

Everyone tries to work out how many soldiers were garrisoning Britannia during the age of the Roman Empire. What is different about what you guys do? To answer your first question, there's kind of nothing we aren't doing with our data. We take a very wide net approach. It's really not that different than traditional history. I mean, there's a ton, as you say, of quantification. There's adding things.

There's counting things. There's a lot of numbers in even traditional history. I would say that really what we're doing that's different is just kind of the scale of the questions that we're asking. So we're not asking about what was the number of hordes in the Roman Empire. We're asking, oh, can we use hordes from all over the world or all over Eurasia to indicate and sort of give us a measurement of the level of conflict over like a thousand year period? We're not doing anything different. It's just kind of the questions are a little bit bigger, I would say. It's just kind of the questions are a little bit bigger, I would say. Yeah, coin hordes is a very good example because it allows us to quantify the amount of violence that occurs in societies.

Just think about it. People put coin hordes down for safekeeping during the times of trouble. And then if something horrible happens to them, they get killed or maybe driven into exile or enslaved. Then the coin horde is left in the ground. And so you have these deposits through the history which tell us what were the violent times and which times were peaceful times when you don't find many hoards. So this is a very good use of what you call proxy. It's something that you want to get at, but we lack direct quantitative measurements of,

so we use indirect indicators. And so coin hoards, wherever any societies that had coins, they have coin hordes. So we can compare them not only through time, how violent or peaceful they have been, but also across different nations, across different countries. Do you set out to find some data on coin hordes here? Or do you just scrape data wherever you find it? Like if you come across something with a nice table about the price of grain in the fifth century, are you like, oh, we'll have that? A little bit, but mostly we're sort of theory driven. And so normally we set out with the question first. And so we'll want to know, for example, what is it that exactly comprises what we would call social complexity? So what

makes these large, complex, even ancient societies, what binds them together? What are the common elements they have that are shared across all of these places over the world? And so we'll have theories that try to explain, oh, well, it's the rise of population, or it's the sort of administrative hierarchy that's involved, that's sort of driving it. And there's different theories that point out very specific factors that are involved. And we'll just add those up. We'll sort of try to find all the theories that have been proposed and just number all the different factors that have been proposed as being really important. And we'll say, okay, well, let's find data on all of those factors and see which one in fact holds out to be the most significant.

And so it's really those factors that are theory driven that then informs us. So we have theories of, you know, volatility is an important measure. And so, okay, well, how are we actually going to get that? And as Peter says, that leads us to understand from various work that coin hordes can be a good proxy of that. So then we go out and say, okay, let's find all the coin hordes we can and count those up. Yeah. Maybe it would be worthwhile to step back and talk about the use of general theories in history. Do you think that's a good idea? Because the majority of historians don't really care about general history, and many of them think that they are impossible because human societies are so different from place to place and from time to time. But if you think about it, without denying the huge diversity of human societies, surely there are some common features of them. Just think about it. Agriculture was invented independently all over the world. Societies

increased in scale all over the world. They became centralized, acquired rulers and elites, acquired information systems, writing and things like that. If you think about a weird, bizarre, violent ritual like human sacrifice, that has been also independently invented all over the world. So surely there are some common features. And that's what we who work in this new historical social science called Cleodynamics, that's what we are trying to get at. We are trying to understand why human sacrifice first arises and then tends to fade away. Why do human societies scale up? What is the role of different factors? And so here I come to what Dan was talking about.

So let's say, for example, why did human societies increase in scale, in size, by orders of magnitude, from hundreds to millions, tens of millions, and so on and so forth. And so there is a variety of theories, and historians themselves have proposed many of them. And they are basically opposite explanations. So our job is to use the historical data that historians, multitudes of historians have collected to empirically test those theories and find out which ones actually are closer to the truth and which ones don't really explain the data very well. So that's the major goal of this new discipline.

What do you regard as your biggest successes? What are the things that you like to drive old fashioned historians crazy by going, we think we've crunched the data and we think this bit of received wisdom might be wrong. Come on, tell me, boast. Ooh, that's a good, I'll let Peter go in for some of those. I mean, to me, you know, again, the social complexity work that we did, again, trying to understand what are the main drivers of this scaling up, this orders of magnitude scaling up, finding that it is actually much more uniform across time and space than even we expected. That generally, wherever you're

looking, if it's, you know, in East Asia or in West Africa or Europe or even North America, the way that these societies tend to grow when they sort of scale up in size and population and then get these administrative hierarchies and all the various things that Peter was talking about, they all tend to happen in a pretty common and similar way across the board, which to me, again, you know, as an historian, my background is fairly traditional historical research. That shocked me. Like, that's not the story we tell ourselves. And so I found that very surprising. And I really annoy my classical friends who are, you know, Roman historians with, oh yeah, Rome's not that special. It's not that unique. Don't worry about it. It's

more similar to other things than you think. That's one of my pet hobbies. Yes, that's right. So the first article we published about two years ago in a journal called PNAS, it was the first article that analyzed the data from the SESHAT Global History Data Bank that has been our sort of big project. And I come from complexity science. I'm not a historian by training, but I was also shocked by how, not uniform, but how many commonalities we observe in the evolution of societies from very different parts of the world, from Rome and China, Egypt, Hawaii, Peru, Mesopotamia, and so on and so forth. Another success story is the opposite question. Okay, so we have a big puzzle is how large-scale complex societies

evolved, but the second big puzzle is why do they periodically break down? And so this is the research that we have been involved in over two decades, and we finally start getting very good questions. It turns out that the route to crisis is fairly standard. There are essentially three different components that are involved. Popular immiseration, the falling living standards, elite overproduction, too many elite wannabes fighting for too few fixed number of positions, and the weakening of the state fiscally. So those three factors over and over again, ever since the complex, large-scale human societies have evolved 5,000 years ago, all of them across the world have been breaking down periodically every 200, 300 years, following the same basic scenario with some, obviously, variations, right? But there is something very generic about this. In fact, now we find ourselves in the United States in a situation which is very similar in many respects to what historical societies have experienced before, given that our society is completely different from Roman society,

right? Nevertheless, we still have elites and we still have popular imbecilation and all those factors are actually playing a big role. I mean, traditional historians would probably have agreed with that. So do you see yourselves as backing up often traditional historiography or turning on its head? I mean, most people would say, yeah, if you look at the Mongol, Roman and Chinese history, so Tang dynasties and the Han giving way to the Siu and the Tang. it's kind of like conquest dramatic climate change famine immiseration incompetent leadership slash you know internecine competition leading to collapse i mean that's something that i think lots of historians would recognize the difference you're doing is you're bringing in parallels from different parts of the world to me the difference has always been the scale of the questions you're

asking and there's often a lot of agreement and not only some of these conclusions as you're talking about, but even the methods are often the same as, you know, a historian of medieval China and an historian of pharaonic Egypt are doing a lot of the same things. They're asking a lot of the same questions. They're answering the questions in the same way. Really all we're doing, as you say, is taking that step back and putting it all together and say, okay, yes, we see these patterns. There's commonalities. We are using these, you say, is taking that step back and putting it all together and say, okay, yes, we see these patterns, there's commonalities, we are using these, you know, deep historical scholarship and these historical information. So we're collaborating with historians at every step. And, you know, we're just putting it together and say, okay, what are the actual underlying

structural factors that seem to be driving these commonalities? And why is it that we see these repeated patterns? Like, what's the actual sort of underlying context? And I think that's that step that you don't normally get in most historical fields, but it's really, in my mind, fits very comfortably with traditional history. Well, first of all, I want to just stress very strongly that cleodynamics is not in a position to history. In fact, cleodynamics is impossible without history because no matter how wonderful theories you have, they're worthless without data. And where does data come from? Of course, it is collected by the traditional historians. So leo-dynamics cannot exist without history.

But leo-dynamics can add to what historians do, because historians do propose general explanations. Every time you try to explain something, there is a general explanation, general theory lurking in the background. So let's go back to the Roman history. So everybody asks the question, why did the Roman Empire collapse? That's one of the tropes, actually, in historical research. One German historian, Alexander de Mont, he actually once set out to quantify how many theories have been proposed for the Roman collapse. He counted 230-some theories, and since then, more have been proposed. I mean, obviously, all those theories cannot be correct.

And so it is a strange situation when theories are multiplying, but there is no way to actually cut down on their number. And that's basically our job. This is History Hit. You're hearing all about Cleodynamics right here on the podcast. More coming up. Land a Viking longship on island shores. Scramble over the dunes of ancient Egypt and avoid

the Poisoner's Cup in Renaissance Florence. Each week on Echoes of ancient Egypt, and avoid the Poisoner's Cup in Renaissance Florence. Each week on Echoes of History, we uncover the epic stories that inspire Assassin's Creed. We're stepping into feudal Japan in our special series, Chasing Shadows, where samurai warlords and shinobi spies teach us the tactics and skills needed not only to survive, but to conquer. Whether you're preparing for Assassin's Creed Shadows

or fascinated by history and great stories, listen to Echoes of History, a Ubisoft podcast brought to you by History Hits. There are new episodes every week. Peter also tell me more about Sechat this project Sechat is the name of the Egyptian deity of writing and also of databases strangely enough because she is often depicted inscribing data on a papyrus leaf and so about about 10 years ago, several of us, including Oxford colleagues, Harry Whitehouse, Peter François, and several and many others,

we got together and decided that comparative history is the way to go. But comparative history, as it has been practiced at that point, has been limited because any individual scholar would only encompass so much data in their brains. So typically comparative history is done by looking at two, maybe three, a few numbers. So there's been several comparative histories of Rome to China or Mesopotamia to Egypt. So what we wanted to do, we wanted to do comparative history on steroids. And the only way to do it is to actually involve computers in that, so the knowledge is deposited in them. And so that's how we decided that we should build this data bank.

with historians. So the Seshad project involves, at the latest count, nearly 100 historians, archaeologists, some other scholars of the past, like religion scholars and things like that. So we've been delving into this very difficult process of how we take historical knowledge and then transform it into the data that's useful for analysis. And maybe I'll let Dan actually speak about that part, because he was very intimately involved in this process. Yeah. And just to say again, that what attracted me to it as an historian and coming to this from sort of the opposite end that Peter did, by the end of my kind of dissertation in Roman history, I'm exploring the Roman empire and trying to figure out why the empire was sort of so successful and why it started to crack up in the third century. And I started to ask broader questions like, oh, how unusual is this? What is

unique about the circumstances of Rome in France and say the mid third century CE? Are my answers that are very Roman specific, do they make any sense? Are there larger structural factors going on? And that's what kind of brought me into the realm of Seshad and what Peter was doing. And I, when I sort of found out about the project, I was like, yeah, this is for me. This is fantastic. You know, as we say, so that I can explore, okay, well, what is actually common? And then pinpoint what is unique about each individual society I may be interested in. And so a lot of this is, you know, once I got involved with the project, it was, as

I say, and as Peter documents, it's really recognizing the fact that there has been hundreds of years of just amazing, amazing historical research that has produced just a wealth of knowledge and more knowledge than anyone could ever possibly comprehend, more knowledge than even a team of researchers could possibly comprehend or hold in their minds at any one time, which is, as Peter says, why you actually need computers. You need a computational power to recognize patterns from all the sort of noise and messiness of human cognition. So that really the goal and all we're really trying to do is to say, okay, we have so much information, so much glorious material and data

from all of these different societies. Let's put it together and see what we can do with it. And also being fairly committed to sort of open science. And we want, you know, not just what we can do with it. And also being fairly committed to sort of open science. And we want, you know, not just what we can do with it, but we want to be a sort of resource collecting all this wisdom so that anyone else can come and ask their own questions of various sorts of scales. So it turns out that the third century crisis in Rome was not so unusual. It's common to many of the great empires through history. You mentioned earlier, you started to bring things up to the present day, which I thought was very interesting, because reading your stuff, I'm quite struck by the fact that you tend to stop in the early modern period

where other studies of data are better established. If I was a Ming dynasty bureaucrat, your information would be essentially important, okay, because I'd be able to predict the coming of the Qing, okay. But given that I'm not a Ming dynasty bureaucrat, say you're a Biden staffer incoming, what are the big lessons that you can draw that are useful today? Well, first of all, remember, there are two sort of really major themes. One of them is how complex societies evolved. And the second one is why do they periodically go through breakdowns? So Seshat Data Bank was originally designed to address the question of the evolution of complex societies. And since that started happening thousands of years ago, and by 1800, we had very substantial complex societies.

That's why we focused on that period. The last 200 years is not as significant when you look into the 5,000-year history of complex societies. But of course, we also want to use our discipline to help us get out of the predicaments that we get ourselves into, like what we are seeing today in the United States. And so we have started what we call the Crisis DB, crisis data bank, that is actually codifying a suite of variables for all those historical cases we know about how societies go into the crisis. And equally important, how do they sail through it? How do they get out of it? In majority of cases, the results are not great. I mean, sometimes you have complete societal collapse,

almost invariably, you have some kind of a political violence, civil wars, and things like that. But some societies do manage to get through, right? So the goal of Crisis GB is not only to further test our understanding how societies get into crisis, which, as I said, follows very standard trajectories. But then there's a whole fan of possible trajectories that opens up. And so how do we use that knowledge to help our own society? Unfortunately, right now, we don't have a very strong answer to it. We can offer some partial answers. So, for example, the fact that population well-being in the United States has been declining,

right? So this is both economic. As we all know, the GDP per capita has been growing fairly well over the past several decades. But suddenly, the wages of workers in relation to GDP per capita stopped growing around 1970s. So that has caused all kinds of problems, including something that I never expected to observe, and that's that we now have a Malthusian crisis. We have the life expectancy has been declining even before the COVID-19 crisis. Unimaginably to me, the average heights of several segments of American population have been declining, and that's a very strong, direct biological indicator of decreasing well-being. So one of the big lessons

from history and from this work to the incoming Biden administration is that you've got to reverse. You've got to get back popular well-being on track so that the fruits of economic growth have to be equally divided. They don't need to go only

to the preferable 1%. They should go to equitably to the whole population. How you accomplish this, this massive political process is going to be very difficult. Obviously, especially given the current configuration, political configuration, United States, that's your job. But this is something that you have to start, you have to turn around this country and get the popular vote being back on track. Maybe Dan can add to that. I see him nodding. I just want to, I mean, this is very much, you know, you take the lessons of the US and to us, what is, you know, a little bit surprising, but also makes this so important is that what we're seeing in the US and Canada, where I'm based in the UK, of course,

again, you know, the popular immiseration, the fragmentation, the polarization, the sort of vitriol among political parties and sort of our leaders in these different countries, all the sort of problems that people talk about a lot. These are so familiar to us who have been exploring these historical cases. And yeah, we concentrate on the pre-industrial period, but it's amazing how similar the events that are unfolding right now are to these historical moments. And so this is why to us, these lessons become so timely and so important, even as you say to these incoming leaders, this sort of rising inequality, this lowering of well-being, somehow healing over these sort of political and partisan divides, just stand out as being the sine qua non of any sort of recovery out of these crises. What are your judgments about the drivers of history? Like, should we have been less worried about religious dissent? It's more like it's climate, it's food, it's troublesome nomadic peoples on your northern front. Like what are the monster lessons that will determine the course of history that you guys

have been able to draw from this project? It's this sort of inequality and how, you know, inequality, not only economic, but in access and political gain and political power drives sort of well-being among the general population down and leads to all sorts of infighting among elites. And that stands out as time and time again, as a destructive path to violence. You know, the sort of warfare between states is another thing that stands out as well as being a major driver. I don't want to get into that too much about sort of what to do with it. The one thing I will say is that it's not that history repeats itself just because these are, you know, iron laws that are completely immutable. It's because there are common processes to how societies are structured and how we develop.

But the fact that we can be aware of them, the fact that we can name them, at least gives the hope to me that there is some feedback mechanisms there. So simply being able to identify and name some of these key problems should mean that we are then able to avoid them in the futures.

And so, yes, you can have rising countries and jealousy, but that doesn't need to lead to a major warfare just because it did in Thucydides' time, for example. I can hear the Roman Empire avoiding war with Persia. I can hear various Chinese dynasties agreeing with you about fighting foreign neighbors. And okay, so we avoid dramatic economic inequality. Peter, what are some of your gigantic lessons that we need to know now? So inequality has been very much in the air. You know, the Davos people have even named it as one of the important problems to be solved. But to me, inequality is more of an indicator rather than the direct driver. So the direct drivers are, is that the majority of population in the United States and several other Western

European countries are losing ground. I mean, so that feeds into inequality, but inequality is kind of an abstract thing. Well, more directing is that people are living less and they're living more poorly, the majority of people. So this is what we call in technical terms popular immiseration. That's the real problem. That's what is fueling a lot of discontent. And that is what propelled unlikely candidates

like Donald Trump to the presidency in 2016. And that's why an amazing number of people voted for him. That's one of the big reasons why so many people voted for him even in 2020. So the other side of inequality is that obviously all that extra GDP has to go somewhere. And where does it go? It goes to the top layers of the society, the 10%, the 1%, the 1% of 1% especially is doing very well. And that's troublesome also.

I mean, if you think about the people getting wealthy, what's wrong with that? But the problem is when people get super wealthy, first of all, much of that wealth is actually, I'm not afraid of this word, is wasted on what we call conspicuous consumption. Getting your yacht will be, you know, two yards longer than the next. Commentators have noted that, that wealth, extreme wealth and democracy don't go comfortably well together. When you get super wealthy people who become even more wealthy, they use their wealth to influence the political process. And although not all of them are knights, not all of them are selfish people, some good patriotic billionaires come to mind. But the majority does look out for number one.

And as a result, what the government does is actually driven very much by the selfish interests of the wealthy. And that makes for even worse problems. Paradoxically, the more wealth goes to the top. That actually has another little appreciated effect. So we call it elite overproduction. It's not just that 1% gets wealthier. It also is that 1% becomes first 2%, then 3%, then 4%. But there are only so many positions for which this elite aspirants, as we call them, can vie. So if you know, if you want to influence political process, you either run yourself as president or you fund presidents, but there is only one presidential

position, only 100 Senate seats. So increased number of aspirants for these positions drives much greater competition. And it starts in this competition, very soon goes beyond the healthy. Some competitions obviously necessary. You want to see better people get ahead of poor people. But when competition becomes super competition, then it starts to undermine the cooperative institutions that we have to channel the political process in good directions. And we've seen that the institutions governing our polity have been degrading and breaking over the past, especially for years, at a dramatic pace. So this is, again, this is a general lesson. We see this actual mechanisms that create violence and state breakdown. Well, I like it, guys. I like it. I'm interested in it. It sounds good. Let's tell everyone about your book. What's the book called, Dan? It's Figuring Out the Past, the 3,495 Vital Statistics that Explain World History. It's

everything you need to know. Nice one. If you're thinking about setting up a giant land empire in Asia, you cannot do so without this book. Vital. Vital. If only the last song emperor had had this book by his side, he might have avoided his appalling fate. Gentlemen, thank you very much and good luck with your book. I thank you so much. Thank you, Matt Steph. I hope you enjoyed the podcast. Just before you go, bit of a favor to ask. I totally understand if you don't want to become a subscriber or pay me any cash money.

Makes sense. But if you could just do me a favor, it's for free. Go to iTunes or wherever you get your podcast. If you give it a five-star rating and give it an absolutely glowing review, purge yourself, give it a glowing review, I'd really appreciate that. It's tough weather, the law of the jungle out there, and I need all the fire support I can get. So that will boost it up the charts. It's so tiresome, but if you could do it,

I'd be very, very grateful. Thank you. you