The Jordan Harbinger Show - 420: Renee DiResta | Dismantling the Disinformation Machine

Episode Date: October 22, 2020

Renee DiResta (@noUpside) is the technical research manager at Stanford Internet Observatory. She studies the role that tech platforms and curatorial algorithms play in the proliferation of disinformation and conspiracy theories, terrorist activities, and state-sponsored information warfare. What We Discuss with Renee DiResta: How the anti-vaxx movement, the Tea Party, and militia groups learned early on to market and cross-pollinate their messages across social media to receptive segments of the populace with shocking success. The challenges faced by the US government in trying to stem the tide of social media exploitation by fringe groups and terrorists for the purposes of recruitment, generating sympathy for their causes, and fueling general chaos. How entire networks of disinformation evolve to support and perpetuate interconnected webs of conspiracy theories, pseudoscientific bunk, fringe political views, and propaganda planted by hostile states and bad actors. Why foreign organizations like Russia's Internet Research Agency and China's 50 Cent Party have promoted tribalism in the United States to keep us fighting among ourselves, how it serves their interests at our expense, and why we're so receptive to their influence. Why memes and hashtags are so effective as short-form vehicles to promote simple ideas powerfully across wide demographics. And much more... Full show notes and resources can be found here: jordanharbinger.com/420 Sign up for Six-Minute Networking -- our free networking and relationship development mini course -- at jordanharbinger.com/course! Like this show? Please leave us a review here -- even one sentence helps! Consider including your Twitter handle so we can thank you personally!See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

This episode is sponsored in part by Conspiruality Podcast. You know how I'm always talking about critical thinking and spotting manipulation? Well, there's a podcast that's all about dismantling new age cults, wellness grifters, and conspiracy mad yogis, basically the wild overlap of spirituality and misinformation. It's called the Conspiruality Podcast. The hosts, a journalist, cult researcher, and a philosophical skeptic dive deep into how this stuff spreads, from Project 2025 and the Heritage Foundation's dystopian vision of the future to how former leftists get pulled into far-right conspiracies.

An interesting episode to check out is called Speaking Truth to Goop, where Jen Gunter breaks down the pseudoscience behind the wellness industry in a way that is super entertaining and eye-opening. It's sharp, funny, and makes you a lot harder to fool, which, if you listen to this show, you know I'm all about that. From exploring cults to analyzing our cultural and political landscape, the Conspiratuality Podcast will help you stay informed against misinformation and resist fear tactics.

Find Conspirality on Apple Podcasts, Spotify, and wherever you do. get your podcasts. Coming up on the Jordan Harbinger show. Social platforms, they were not developed to be information libraries and repositories of human knowledge where people go find answers to their financial and health and political questions on those platforms. They were to help you, like, find your friends, right? Find your knitting club. Not your scientific and medical authority for how you're going to treat a disease. Welcome to the show. I'm Jordan Harbinger. On the Jordan Harbinger show, we decode the stories, secrets, and skills of the world's most fascinating people. If you're new to the show, we have in-depth conversations with people at the top of their game.

Astronauts, entrepreneurs, spies, psychologists, even the occasional drug trafficker. Each show turns our guest's wisdom into practical advice that you can use to build a deeper understanding of how the world works and become a better critical thinker. Today, not a drug trafficker, but my friend Renee DeResta. She's a researcher who studies influence operations and propaganda in the context of pseudoscience conspiracies, terrorist activity, and state-sponsored information warfare. Now, if you haven't heard about this, Russia is looking for the cracks and fissures, and then once, of course, they see something, they ignite and pour gasoline on the fire.

And I just mix that, well, whatever, Putin loves a good mixed metaphor from what I hear. You get what I mean. You may have also noticed that online arguments get nastier and more quickly because the trolls are escalating things. Now, of course, it's important to separate Russian interference with collusion and things like, well, Trump wouldn't know one without this. That's neither here nor for this conversation today. The above really clouds us from looking at this objectively. Russian election interference happened. We know this. Intelligence agencies say that it happened. The government says that it happened, but the platforms themselves have seen that it happens. Everyone acknowledges that it happened and is continuing to go on around the world.

It's not the same thing as collusion between parties and Russia or whatever, for those of you who are just bristling at this or just not sure of the difference. Now, today, we'll discuss the extent of these influence campaigns and how they work, including why we see so many bots and real people alike, spewing hate online, as well as how platforms like Facebook were used as pawns in the plot to create strife and division right here in the United States and Western countries in general.

We'll also explore some practical ideas on how to protect ourselves from this, including how we can tell if we are seeing an influence campaign or if it's real grassroots activism and why Americans have to be especially vigilant as our 2020 election draws closer. If you're wondering how I managed to book all these great authors, and celebrities every single week. It's because of my network. I'm teaching you how to build your

network for free over at Jordan Harbinger.com slash course. And by the way, everyone you hear on the show is either contributed to the course, is in the course or both. So come join us. You'll be in smart company. Now, here's Renee DeResta. How did this get on your radar? I know this sort of started with you trying to do The Impossible, which is fine to preschool for your kids. Yes. So I had my The first son in December 2013, and in California and San Francisco, you have to get on these waiting lists, like a year ahead of when you want it. So in December 2014 was when I had to get them on these lists. Anyway, I started Googling around looking for schools, but also I wanted to look at the vaccination rates because that had come up. I used to live in New York and it was kind of required, but in California you could just opt out because of personal beliefs.

You could just say, we're not going to vaccinate and that's it, philosophical exemption. So I didn't want to go to a school like that. And I downloaded a bunch of data and was really disturbed by how the rates had been really trending down over time. And I wrote a blog post about it late November looking at like how private schools were a million times worse, how Waldorf schools were an absolute disaster, just trying to say like California parents, like we, you know, we should be demanding better here. I called my local representative. I'd never done that before, but I felt like that's such a Karen move. I was so annoyed. I was like, come on.

Like, how is it that there's like, no, there's school like 30% immunization rates? And I was like, what the hell is going on here? Like, can't we do something about that? Yeah, and they were like, yeah, no, nothing to do. Nothing to do about that. You know, good luck. And then the Disneyland measles outbreak happened a month later. And I called back again.

And I was like, hey, how about now? And they were like, actually, yeah, you know, we have a senator in Sacramento who's going to be kind of introducing a bill to deal with that situation. And I said, you know, I'd love to help. Maybe I could, you know, do some data analysis for you guys. I'd love to be useful in some way. And so they put me in touch with two other moms, and we started this group called Vaccinate California. I started a group, I mean, we made a Facebook page, just to be clear.

Yeah. There was no legal filings. There was nothing that this was going to be like a big deal. We didn't really know what we were getting into. So we just made a Facebook page. And I started running Facebook ads because we needed people to call and to support people. And so I started running Facebook ads. And I got, you know, I learned how to do ad targeting and look alike audiences and a bunch of other kind of

more sophisticated ad targeting tools. And in the course of doing that, we also started putting out content muckabill on Twitter. I was like part of SF tech, you know, been on Twitter for years and was kind of completely blown away by like activism, Twitter, particularly by how well organized the anti-vaccine movement already was. And the way in which these fake accounts would kind of come out of nowhere, they would say that they were from California, they'd been created yesterday, they were communicating, they were being given instructions on YouTube videos that were being put out every night, saying these are the hashtags you're going after, here's the memes you're going to use, here's the folder with the memes in them. And I was kind of like, that was a really amazingly

organized operation. On Facebook, what I started noticing was if you searched for vaccine for running any of your ad targeting, what it started giving you was anti-vaccine suggestions. So if you typed in vaccine, it would push you a bunch of anti-vaccine topics and you could target anti-vaxers really easily. But I was like, well, how do I find, like, parents who want high immunization levels in their schools? And then I realized that there was, like, this asymmetry, right? Like, most people vaccinate, but they don't stop to think about it. They definitely don't become activists about it. They vaccinate their kids. Nothing happens. They move on with their lives and, you know, and they don't get on social media to talk about how nothing happened. So I was kind of struck by this asymmetry

and by saying, like, okay, how do we rectify this? Because it seems like all of the algorithmic boosts are driving things in the other direction. And so I started looking at mapping the network. So I did some network analysis with a data scientist named Galaglo Tan. And we started looking at how the anti-vaccine parents on Twitter, who had been anti-vaccine for years and years and years, were beginning to reach out to the Tea Party, for example, to some of the militia movements, to some of the Second Amendment activists, all of which are, you know, again, like kind of communities that had been on Twitter for a really long time. I haven't heard about the Tea Party and like. I know. This is the Tea Party in like 2015, you know, like late 2016.

early early 2015. Yeah. It really was. So this was before the 2016 campaign. This was all happening in January to June 2015 or so, so pretty early on. And I started writing about how social networks were enabling different kind of persistent factions of people, different communities, to find each other and to find, you know, to kind of like marry hashtags was what the anti-vaccine kind of activist called it so that if you wanted to push your anti-vaccine content into Tea Party communities, you would use the Tea Party hashtags alongside yours, and then they were kind of using these basic marketing tactics actually for policy activism. And I thought that was really interesting because it made me feel like everything was going to be a marketing campaign for an idea. Any policy that went out the door

from now on was effectively going to use these tactics. And so the people who were best, most adept at deploying them were going to be the groups that got their message heard because what mattered was share of voice, right? On social media, what matters is how much of your audience can you reach? can your meme become the sticky thing that everybody says? Does it have virality potential where it's going to be forwarded along? How are regular people going to see your content and engage with it? How are influencers going to pick up your content and engage with it? And so I started looking a lot at the dynamics of just this really one niche bill in California that was highly politically polarizing. There was a ton of media content going out about it. I got dached. I got harassed. I was not expecting that either.

Sure. Yeah, of course. You know. That was a surprise for you at that time. And now it's like, oh, of course. Now it's like, obviously, right? Yeah. So also the cost of being a public advocate has really changed now, too. You know, I hear I think I'm just like a mom.

Right. And that's not it. Doxed, by the way, for people that don't know is like they put all of your personal information online, essentially to say, call Renee DeResta at her place of work and at home and, like, bother her and her family because she's doing something bad. I don't think everyone knows what doxed means, but it's worse for people who aren't already public figures, because, of course, someone can just find your office and call you. You work wherever you work. But it's bad when you're an activist and you have a bunch of crazy people calling your husband's workplace

and telling him nasty things about you, which is kind of the point of doing that, right? Right. And so I used to have a little Tumblr blog where I put up pictures of my kids and, you know, like, well, I had one kid at the time, but, you know, wedding pictures. Like, just your life, the way that normal people engage on the internet, locked all that shit down. Yeah. Took it all down, you know, sort of like, you know, Twitter's response to me when it happened was like, effectively it was like, you're a limited purpose public figure now, like, good luck to you. And they started making YouTube videos about me also. And I was like, I'd never heard that term before. I didn't realize that. What does that mean? Limited purpose public figure is a legal designation

that says that you have chosen to inject yourself into a particular narrowly tailored conversation, which for me was immunization policy in California, which meant that I was a public figure in the context of that topic that I had chosen to engage on. Now, legally, that would mean if I were to file a defamation suit, I would have to, you know, prove an intent to harm as opposed to just, you know, intent to harass, which is protections are different for these different tiers of publicness, I guess. Yeah. And I thought that was bizarre too, because I thought, well, on the age of social media, anybody who, again, like, throws up a Facebook page, puts up a Twitter account, you know, and weighs in on a conversation, if the internet gets mad at them and turns them

into a public figure, like, boom, there you go. You know, you're... Right. You don't have a choice. Like, before it was you had to run for office, that's one of the sort of traditional public figure occupations that's listed in the law. It's been a long time since law school,

but I think the other one would be, like, if you are an actual celebrity. Right. Like, regular people would recognize you. Right. And so you have all of the downsides of being a public figure, but none of the real upside of, like, getting money thrown at you to wear Chanel perfume

and, like, getting tables for your restaurant. Or your bodyguard, you know? Right. No. Downsides only. Right. So I had this experience and I thought like, all right, so I learned a couple things here. Like we did get the law passed. So, you know, success. And I was a really hard slog. I was very, very proud of being able to get that done. We were the first state to do it in something like, I think, got 30 or 40 years. And the law to be clear was what again? The law said that you have to have a medical reason to opt your children out of vaccines. That was it. Gotcha.

You had to have some sort of like medical justification as opposed to I don't want to. Right. Yeah. You know, so, I mean, there are reasonable people who can disagree. on whether, you know, how they feel about that law and where they think the line should be and what the state's obligation or responsibility should be. I feel like reasonable people shouldn't turn that disagreement into doxing, harassing, threatening, and, you know, that's where we kind of get into the realm of like what the Internet delivered as far as policy conversations versus the olden days where you'd fight it out through some op-eds in a newspaper, right? Right. So I learned a lot about that. I really felt like I got this introduction into network activism. And I think it was the fact that we took a very quantitative approach to understanding how these groups were operating, how they were networking, how social media algorithms, how curation and amplification were really amplifying this content.

And I had been a venture capitalist. So I had done a lot of investing in tech companies and looking at tech business models, looking at implications of technology on society. And I really felt like what I was seeing was this shift, this thing that was going to fundamentally change political campaigning, policy advocacy. activism, online in a very, very fundamental way. And then what wound up happening for me was the White House reached out. And through Todd Park, former Chief Technology Officer, he asked me to meet up and he said, you know, some of the work that you're doing were trying to do some similar work looking at how terrorist organizations are using social media for recruitment, advocacy,

and propaganda. We think that the way that you've been writing about this and talking about this and, you know, seeing this develop is something that we'd like to have you kind of come down to D.C. and participate in some of our conversations about this in the context of this other group. So that was not, you know, really where I expected my career to go, but that was where it went. Yeah. I went to D.C. for about three weeks, worked with a group of people who were, who had done a lot. You know, I was not a counterterrorism expert, but this was ISIS, of course, was what was going on at the time. And if you recall back in 2015, this was the era of the beheading videos. And the ISIS fanboys,

the recruitment process, recruiting the women to go be jihadi brides, looking at ways in which, again, if you followed one jihadi on Twitter, Twitter would suggest like six more, you know, looking at ways in which kind of radicalization was happening. And in a very real sense, not in the debatable is this radicalization or not, like an actual international terrorist organization was using this platform to push out its message and recruit new adherents. There were what they called ISIS sort of fanboy.

So there were the core accounts that would occasionally post the beheading videos and things, those would come down quite quickly. Twitter would take those down. But then there were the people who would try to serve as like the amplifiers or the people who were like, hey, Twitter is censoring this content, but you can go find it over here on this other platform directing people to things like LiveLeague and other places. So really working to amplify the content, amplify the message. And the thing that we started talking about was what happens when social platforms became kind of vast tools for propaganda? What happened when they became vast amplification tools? that really in a very different way than propaganda had been done in the past,

it wasn't being carried out only through the media, it wasn't being carried out only through broadcast or print. It was a participatory process. It was ways in which you could produce content, a meme, a YouTube video, you could go make your own content, and then you could disseminate your own content. And what was happening was these very small organizations were able to follow that process, put the content out,

and then real people would come and serve as amplifiers, leaving this question of if you were to take down or delete the accounts of the people who were amplifying it, were you kind of stifling free expression? So this was where that kind of like laundry, the manufacturing process of one group producing the content, but then regular peer-to-peer dissemination was something that we'd not really seen before. It was something that the internet really enabled in a way that prior broadcast mechanisms and print just hadn't, you know, it wasn't really possible. And this took Washington and everyone, surprise, right? They were thinking like, oh, well, if we find a channel that has ISIS propaganda,

we'll just close the channel, we'll just close the website, we'll do something to that. Now, when it's like thousands of users or tens of thousands of users on social media, you have to invent software to track it. And then if you shut that, there's all these free speech issues, right? I assume that they then said, there's always like this slippery slope fallacy where it's like, well, if we shut down anti-vax, then what else are we going to shut down? And I would imagine they use that with ISIS, although what's the argument? If we shut down, beheading videos, then what's next? No, it was one man's terrorist is another man's freedom fighter, and if the U.S. government requests be taken down, what if a more authoritarian government

requests take downs in the future? That argument did come up, actually. The slippery slope was very much a part of it. There were a couple interesting questions, right? One was, first, it was not a surprise to them. Actually, there's been a DARPA program in 2012 called Social Media and Strategic Communication, S&I-S-C. And because the DARPA program is open, anyone can actually go and read the archives. So anyone who's interested and go read the research that came out of Smisk. And DARPA is our defense research

kind of like invented the internet and looks at technology in terms of how we can, it can either be weaponized against us or how we can weaponize it. Is that kind of a layman's overview? Yeah, that's a good description. I think the motto is to prevent

and create strategic surprise, I think, is the, basically how you both prevent it and put it to use at the defense department, defense advanced research project. Agency, agency, there we go. I don't see a motto anywhere, but I haven't looked that hard.

Defense Advanced Research Projects Agency. That's what that is. Okay. So DARPA had this program from 2012 to 2015 called Social Media and Strategic Communication that actually asks this question, what happens if hostile governments, terrorist organizations, you know, unsavory characters begin to use the Internet for these kind of nefarious purposes, particularly propaganda and spreading false and false. So it was not a surprise. I think the question really became, what could the government do about it, though? And there were certain limitations in place. So at the time, the State Department could counter message, but because of U.S. law that says that the government is not allowed to propagandize to its own citizens, the State Department messages that went out over Twitter, if they might be seen by an American, had to be declared and attributable to the State Department, which meant that the entity that was kind of tweeting back at ISIS sympathizing.

on Twitter was this U.S. State Department group. So it's sort of like, well, it's not like the U.S. State Department is really going to defer somebody who's on their way to down this radicalization path becoming an ISIS sympathizer. So that's not going to be the most well-received counter messaging source. You're not going to deradicalize somebody with a State Department tweet. It's like the opposite of influencer marketing. It's like, we want influencers to wear our cool shoes and show how useful they are. And then it's like, please buy our shoes.

Like, no, no, you're not going to be. away. The hashtag that they were using was like think again, turn away. Oh, brutal. It was not the most well executed. No. Early kind of counter operation. But they recognized at least that they had to do something, right? So there was a widespread acceptance of the fact that this was going to be sort of like the new normal that information wore through terrorists. And at the time, as we were having these conversations in October 2015, we also already knew that Russia was operating on U.S. social platforms. That also was not surprised. The extent was a surprise. The extent was a surprise. specifics were a surprise. But at the time, as we were thinking about what would the U.S. government's response to this kind of activity be, what should the tech platform response to this kind of activity be, even in the very narrowly scoped realm of counterterrorism, the question really became, well, if regular domestic like moms in California can do it, and if ISIS can do it, better believe that, like, Russia and China can do it. Right. And so it turned into a little bit

of like, well, maybe the kind of counter messaging entity and the whole of government response to this stuff shouldn't be some random State Department Twitter account. Maybe we should be forward looking and think about the fact that this is, I couldn't tell if it was like Chicken Little or Cassandra, you know? Right, right.

Something is changing here. Like, I really feel like we should be giving this some like pretty big guns in terms of get a lot of like the best brains in the country thinking about what's going to happen here because it seems like it's going to be a disaster. And yeah, and like I said, so Adrian Chen's article, about the internet research agency had come out six months prior.

The article called The Agency in the New York Times. So again, the fact that Russia was running fake Facebook pages and doing these things was not news. But the real question became like, what happens when it's all happening on a private platform? So the government doesn't really have that kind of jurisdiction. You know, what are the regulatory mechanisms? How do you compel a platform to take action if you choose to do so?

So let me pause for a second, because I think a lot of people, they look at Facebook groups, and there's an algorithm that's doing some of the work, right? It's not necessarily just information warfare actors from Russia and China. Right. It can be, if I join a group that's anti-vax, and I've done that before to see what's going on in there, especially when I do shows about that kind of thing, it's kind of a quick jump to like QAnon, California, right?

And like Pizza Gate, you know, truthers, 9-11 truthers. Like, those are the things that were in the sidebar at the time. It wasn't like biking in the Bay Area, right? It wasn't normal stuff. You search for COVID-19 or coronavirus. It's not like coronavirus information network. It's like coronavirus truth, coronavirus warriors, coronavirus, whatever. Like, it's misinformation rabbit hole that you can't, you almost can't even find the real stuff.

Yeah. The algorithm is just like, oh, you believe dumb crap? Here's a lot more dumb crap for you to deal with. So there's a reason for that. So when you build a social network, particularly if somebody's onboarding, you want to show them things that are relevant to them, which means that you build a recommendation system. And before it has data from the person, before it's instrumented the user and follows their clicks and what they engage with and what they do, you're kind of a blank slate. And so what it does is it does one of two things.

It shows you accounts that are geographically relevant. It keys off the information it has to suggest interest based on certain demographic characteristics, certain things that maybe you have selected. But the other thing that it does, so when you engage with content, it can show you more content similar to that type. So that's called content-based filtering. So it says, you like gardening. Here's 50 more gardening pages. Then there's the other thing which is called collaborative filtering, which says you in its aggregate and its understanding of who you are

based on your clicks and your behavior and your demographics and all of the data that it amasses on you, if you engage with this type of content, people who are statistically similar to you who engage with that content, also engage with this other content. So if you're engaged with gardening, maybe you're also a cook. So even if you've never searched for cooking, we can push cooking stuff at you because 75% of people who like gardening also like to cook.

An oversimplification, but that's basically how collaborative filtering works. So it says that some percentage of people who like this thing like this other thing. So what happens with conspiracy theories, there are some people who are into the anti-vaccine conspiracy, for example, because they have health, you know, they're concerned about toxins or they're concerned about the sort of purity people. the, you know, my baby is born perfect. Why would I inject the thing into them? But then there are other people who are involved in the anti-vaccine movement because they believe that there are vast government cover-ups and the New World Order and the Illuminati and Farma and these sorts of, the people who are more on the conspiratorial, someone somewhere is harming people and I'm not going to be a victim of that. Those people are more likely be receptive to other anti-government conspiracies or conspiracies that are alleging that evil nefarious horses are at work. And that's where you start to see. even if you originally go to the anti-vaccine group because you are a natural health like juice person. It still says, well, of the 10,000 people in this group, 7,500 of them are also members of the Pizzagate group,

ergo we're going to serve you to Pizagate stuff too, even if you've never gone and seen it. So I also started getting, you know, I had joined a bunch of anti-vaccine groups and pages when we were doing the vaccine law. And so I had this account that was regularly, you know, I was getting recommendations for Pizagate and for Q&ONON. And so that tie, that kind of conspiracy correlation matrix kind of became pretty clear really early on. And that was another thing where we were trying to flag that and say, hey, maybe, maybe this is one reason why, you know, some radicalization is happening. Maybe pushing ordinary people who have never gone looking for Pizza Gator Q&O and like pushing it to them and saying, hey, you should see this, you should see this, you should see this. Maybe that's not the most kind of ethical design for a recommendation system. Maybe that's not how we want recommendations of things to work.

Yeah, remember when we thought cyber warfare was going to be like attacks on power plants and infrastructure and now it's like your gardening group suddenly links you three groups later and you're talking about how essential oils can cure COVID-19 or drinking bleach can do it. And it's like, unfortunately that algorithm has sort of mastered finding people who lack critical thinking skills and then like pushing their boundaries a little bit, little by little. And even people who are otherwise sensible can have their boundaries pushed like. this, especially if it's done over time and in a calculated way. And it's like this algorithm that's designed to serve you a bunch of ads and keep you on the platform for as long as possible,

just happens to also be really good at that, good at pushing people's belief systems. So it's not just the platforms. It's not just the algorithm. You mentioned earlier the internet research agency. I don't think IRA has ever been a good acronym for anything. Not a great track record for that one. What is the internet research agency? Sounds so innocuous. Sounds like an academic think tank. in Russia attributed to an individual named Evgeny Progoshin. Progosin is an oligarch, a Russian businessman, with close ties to Vladimir Putin, president of Russia. And the Internet Research Agency began operations in around 2014. I think late 2013 established 2014 operational. It originally operated largely like a comment farm, kind of leaving comments. And then as Twitter became a bigger thing,

tweets on content related to topics of interest to the Russian government. So that originally started locally. So Ukraine, the annexation of Crimea, MH17, the downing of the jet, the Malaysia Airlines jet. Its goal was really to kind of nudge the conversation in particular directions, to distract, to do the things that you would do. You know, China had actually set up an operation called the 50 cent party, sometimes called the 50 cent army back in 2004. So again, there are certain entities and certain governments that believe that kind of controlling the narrative first domestically, first within your local kind of sphere of influence, the internet research agency rapidly expand ended outward. And by mid-2014, early 2015, was actively targeting Americans as well, American

citizens. Yeah. So it didn't start as an entity to disrupt the presidential election of 2016. That, of course, is the thing that it's perhaps best known for because of, you know, the investigation and the, you know, that's been a very much a topic of conversation, Russian interference. But it did start several years prior to that. And it was a multi-year operation. One thing that's kind of remarkable about it is how it took very old. active measures tactics, things that had been carried out in the Cold War, the Soviet era. And I think the real insight was that propaganda was going to shift with the shift in technology. And again, this was something DARPA saw too. Russia executed on it and prioritized it, actively prioritized,

it actively made it a strategy, actively made it something that it put to use for geopolitical influence operations. And so that was, again, that recognition that propaganda evolves to fit the information ecosystem or architecture of the day. The new information architecture was social networks. The new information architecture, the means of transmission was peer to peer. So rather than executing simply on old Cold War propaganda tactics of placing an article on a newspaper and kind of laundering a narrative up the chain of newspapers, they realize that you could make a meme, push the meme to people who would be receptive to it, and then have those people act as your disseminators. And it would become this process by which content designed

for the internet, designed for people, very much like an extremely modern social media marketing approach to propaganda was what this entity really kind of took and ran with. And they're trying to create tribes, right? These aren't just like meme posters. I know some of the earliest pages were that they created these like Facebook pages were for the religious community, for the right. And then they kind of switch morph whatever to black activism. And then it's like Texas Pride, LGBT Pride, Texas Secessionists. And they're like, targeting these in-group communities, why are they doing this? What's up with the tribes?

And it seems random to somebody like me looking from the outside or someone that's new to this, looking at LGBT Kansas pride. How is that possibly related to somebody trying to change an election or trying to change public opinion? It seems so niche. How are the 500 people who are LGBT in Kansas going to do anything? Like, what's the strategy here? So the internet is really made to help you find people.

who are like you, right? So much of that is built into the design. It's prioritized. It's actively prioritized. This is why Facebook developed groups. You used to go to Facebook with your real friends, and you would see what your real friends were putting out there. But if your real friends, you know, either weren't that interesting, weren't that active

on the platform, for the platform to keep you there, coming up with the structure of groups is a fantastic way. We help you find new friends. We help you find new interests. And particularly in a, you know, kind of increasingly polarized U.S., a lot of that was people who would come to Facebook. Facebook to kind of like fight the culture wars, right? You know, they find their political group and get entrenched and become, you know, really active. And you started to see this on Twitter as well. Again, these kind of like persistent groups of people who were really aligned around a particular shared interest. You had your internet friends all of a sudden. Right. And sometimes, you know, for people who are more into like fighting and trolling, you remember the 2015, 2016 campaign, right? Real Americans are just on there with they like they put their Pepe the frog or they. their blue wave or, you know, whatever.

Like, you signal now it's like the DSA Rose, the neol liberal, like, globe. You know, whoever you are is like, boom, it's like right out there. And, you know, your affiliation, you're just wearing it on your, uh, your little, like, username line there. It's the MAGA hat of your social media account, right? Like, there's even lobsters. Those, I don't know. They're like a Jordan Peterson thing.

Oh, yeah. The Rose is socialism. I don't understand why those are that way. I've asked about the rose and I got an explanation and forgot. But yeah, like, then there's the blue hat emoji. emoji. So anything that you can possibly imagine. The avocados for California, you know, kind of of making fun of the idea that if you just eat less avocado toast, you can buy a house. Oh, yeah, there's something to that, though, actually. You can at least pay your mortgage with that.

So we're wearing this on our sleeve. We're fighting with each other in, like, you can even just find people to mess with by searching for that particular hashtag or that particular emoji, right? You can just pick fights that way. Yeah. And so you think of the internet as like a networked series of factions, right? Just this kind of war, all against all. Everybody's out there trolling. What the internet research agency does, their big kind of innovation is they realize that they can create Facebook pages and they can pretend to be members of these communities. And so the content that's put out is 90% of it is really about solidifying that in-group dynamic, right? It's about saying, I as a Texas secessionist believe this. I, as a Black Lives Matter activist, believe this. And so the

The point of the pages, what they're doing is they're really instilling pride in that group. We are descendants of Confederates. You know, this is not about racism. This is about our pride and our Confederate heritage. So most of the narratives, even for the groups that many Americans would see as like highly polarizing, was reinforcing that in group pride. And then when you do that, there's a kind of a presentation of other groups as the other, right? So the question then becomes, if I, a real American, have this identity and this other guy's identity feels kind of counter to mine, the question that kind of was sort of underpinning a lot of it is like, who is America for? So if you believe in a, you know, kind of like there's a finite set of resources, why are we giving money to refugees when we should be supporting our veterans?

And that was a theme that was like hammered home constantly. Why should black Americans vote if this country has never done anything for them, right? And so for us is how it's phrased. it's always phrases like, I am a member of this community speaking to people who are just like me. It doesn't read as like media from on high or kind of like ivory tower intellectuals or media pundits talking to you. It reads like you're having a conversation with people who are just like you. And that's the, so there's that element of you're receptive to the message because it's coming from somebody who is theoretically just like you. Right. But unless you're like a Russian 20 something,

30 something that lives outside of Moscow, they're not really. really just like you, they're not really just like you. Right. Why the focus on the black community specifically, or was that just an example you picked? No, no, no. That was actually a huge percentage of the content was focused on the black community. So the Senate requested from the social media platforms, from Facebook, Twitter, and Alphabet, Google, which included YouTube, that they provide all of the data related to this operation. And then I led one of the teams that analyzed that data set. And there was another team that did the same analysis with the same data. We were told not to communicate because they wanted to make sure that they could check, you know, make sure that

different teams found the same things because this was such a political live wire at the time. This was 2018 that we were doing this work. It was before the Mueller report came out. And so what we got when we were given this data set was this look about, you know, 400 gigs worth of stuff, several hundred thousand memes, 10.5 million tweets. So just this kind of corpus spanning multiple years of their activity. And then what we were tasked with was saying what are the kind of tactics techniques and procedures that this particular adversary used, what were they doing, what was the goal, what were the messages, how did it work? And you couldn't look at it and not see the extraordinary effort put into expanding racial tension. Now, when you say

memes, I think a lot of people know what those are, but when I think meme, I think Kermit the Frog sipping tea, I think somebody Rick rolling me when I'm clicking on a link, right? Those are what I'm thinking of. Are we talking about the same thing? Yeah. So meme, the kind of geeky academic definition goes back to Richard Dawkins, right, the idea of a cultural gene, a unit of culture. And much the way your genes kind of in aggregate form you as a person, a meme is something that is intended to be spread, it's intended to propagate. It's not only the kind of square cap picture with the white words on it, which is, I think, what has come to be thought of. Okay. It's things that sort of signal of participation in a particular community.

If I were to say this is fine, probably maybe 50% of your audience would immediately see the dog on fire room, right? You know, if I were to say winter is coming, maybe you get more people, right? Yeah, this is your brain on drugs? Is that one? That's something I remember from the 80s. Right, right, exactly. So basically anything I remember from the 80s or 90s is potentially a meme because

everything else is gone. That's how effective these things are. Yeah, well, they're intended to be sticky. They're intended people will. use them in a certain way. If you were to go and look at like the K-pop community on Twitter, I am not a K-pop fan. I'm not a not fan. I just don't pay much attention to it. Not yet. Yeah. You know, there are these like hashtags. There's like a whole vocabulary, right? There's like a little like in-group language of ways in which people who are part of the community

talk to each other. One of the things that's really unique about the internet is ways in which searching for certain hashtag, certain phrases, certain words kind of like is the gateway to finding that entire faction, the entire community. Hashtag BTS, right? Right, no, BTS is up. Their new track is fired. But what the internet research agency did, if you look at their Instagram posts, every single Instagram posts has like 40 hashtags down at the bottom

because they know that that's how people are going to find them. So if you're searching for, you know, in the most basic form, hashtag maga, right? You know, was the one thing that the Trump campaign did very well was their slogan became, you know, really, it could be reduced down to an acronym, and that acronym became kind of a meme. It became a thing that people would use to signal their support by putting the hashtag in their Twitter bio by putting the hashtag on Instagram. So you make your community discoverable in that way. And the other thing that's really interesting about memes, they don't require very much thought usually, right? It's something that kind of immediately hits. It's got some emotional resonance.

It's funny. It's pithy. It's tailored for kind of internet style communication. The old and is of propaganda, a lot of times it was like kind of long-form narratives. You would read a persuasive article or an article that made an insinuation. You would feel a certain way, maybe, but it required a lot more time with the medic propaganda with just the, you know, I believe that veterans before refugees, like and share if you agree. That's all you have to say, right? And you've communicated information about yourself, information about what you believe, a political point of view, like and share, that takes two seconds, that there's no real heavy lift. You're not asking if someone could read a thousand word article, they click a button and it's moved on to their

network as well. And so that's the kind of propagation that happens. So it's like virtue signal plus propaganda in an easily snackable, shareable piece of content. Yeah. And to be part of the community, to be part of the activist faction, all you do is click the share button. Are these things being created by bots or real people or both? Because I'm always on the fence, right? I do a lot of stuff about the Chinese Communist Party and like organ trafficking or whatever. And you see the Wumau, the 50 cent army. They'll post. And some of it is clearly just like an automated thing because they've done zero looking into anything that I've done. Or I'll get DMs when I do stuff about Russia. Like I'd Clint Watts on the show. And he was like, watch out. You're going to get a ton of

Russian bots and Russian hate mail. And I was like, eh, no problem. And my DMs were just alive with people who are like, you should have been an abortion, like stuff like that, like horrible things. And I was like, oh, this is just automated. I bet this person tweets it to everybody who is on their, like, hate list. But then other things look like real people. Other things, like someone will engage and some of them seem to just be really sort of dumb folks from wherever. But other times it's clearly a foreigner because they're like, what is it called? Like subject verb agreement is off and everything.

Like the punctuation is off. Yeah, the articles are wrong. Pronouns are wrong. Stuff like that. So you read an interesting thing there. So I'm at Stanford Internet Observatory now. I look at both China and Russia, and we've actually done quite a lot of work on China recently. Because one of the really interesting things, you know, well, the Internet research agency,

I believe Russia is still kind of most effective, most sophisticated at understanding how to target Americans. You asked earlier why racism, that was a very common theme in the Cold War also, right? How can you say you're a free society if black people are treated so badly and don't have rights, right? And the sort of civil rights era. They have a very kind of deep bench to work with on understanding. what messaging works with Americans. As we've seen China come into the game, as we've seen China expand from the Womau, which focus on the domestic Chinese internet into how do you execute those operations targeting people internationally? And that's a really interesting question.

So what we've seen from China, first of all, propaganda is a core part of Chinese government public diplomacy. So there is a propaganda bureau. This is not a thing that is done surreptitiously or secretly, it's quite out there. So they have a large state media apparatus quite well developed. That's attributable. It's sometimes called white propaganda. The attribution is quite clear, as opposed to what Russia was doing, which is sometimes called black propaganda, because the attribution is nebulous. It's kind of opaque. It's hidden in the shadows. It's actively misattributed as opposed to knowing, you know, where the provenance comes from. So what we've seen from China is, first of all, they've taken their broadcast apparatus, their television stations, their radio stations, their print news,

and they've begun to establish presences for all of those other publications on social platforms as well. So they have hundreds of millions, not an exaggeration. I think CGTN, China Global TV Network, has over 100 million Facebook fans on its page. Wow. Yeah, I mean, to put that in perspective, I think CNN has about 33 million, right? So this is sort of like 3x CNN. Rush it today has something like between 7 and 3.3.3.000. 15 dependent. They all have regionalization for their channels. So what you see is the overt apparatus courted to Facebook, even though Chinese citizens are banned from Facebook, they can't go on Facebook. The purpose is not to reach the domestics, the way you have with some of the Chinese

state media and the Murmau, it's to create ways of reaching audiences globally. And so they run ads. And so you'll see CGTN boosting their content, for example, about coronavirus. They began to run ads kind of in earnest, pushing out the Chinese state position on coronavirus where it had originated, how the country was handling it. All of the kind of glowing stories about, well, China sent PPE to Italy, China built this hospital, you know, so these sorts of positive stories. So they're using the Facebook apparatus as yet one more channel in an overt propaganda strategy. Then the other thing that they do is they begin to run bots. And that's where you get into, again, that actively misattributed. these are just ordinary people on the street talking.

But again, they are controlled by the attribution is made to the Chinese Communist Party. It's a little bit difficult to make an attribution to like where in the Chinese government this is attributed to. So Facebook and Twitter say kind of China or CCP. But what we see from those accounts is, as you're describing, it's very much like the Wu Mao kind of ported to this other platform. It's not very well targeted. The personas are not very sophisticated. It's not like Russia where they had these persons. personas that were so convincing that real people, real big influencers, Donald Trump Jr.,

Janusz, D'Souza, Jack Gorsi, were retweeting these accounts because they really seemed to be a black woman activist, a, you know, marvel or man style like, you know, Trump supporter, right? And so where Russia invested years in developing these personas and making them convincing, What you see from China is this kind of throw spaghetti at the wall. The accounts don't even have plausible names. They rarely have a profile picture. They don't have a great bio. It's just crap.

Yeah, it's like an anime or like a stock photo and it's cropped poorly. Right. Yeah, it's almost like even the creation of the photo was automated using like basic AI and it just didn't work very well. It's like a screenshot from a photo of something else. Like it just, and the name is like Huggy Bear Rose. And you're like, this is not a real name. It's a bunch of random words that sounded cute put together.

Somebody who didn't speak English fluently thought sounded like a name. You're absolutely right. You see this. They're not very persuasive. But one thing that we're looking at right now is trying to understand the motivation for that. Like, why would you run an operation that's just such garbage? Like, what is the point? Because when you go and you try to take over a hashtag or put out a hashtag or do something like this,

Twitter is watching now, right? There are integrity teams that are designed to find state sponsored operations on the platform and this is the kind of legacy, you know, the improvement post-2016. So the question is, like, why do you do it? And one of the things that we're thinking about at the observatory is not all propaganda is designed to persuade, not all activity is designed to persuade. Sometimes it's designed to just make it too hard to find the good stuff. So if in the case of, like, the Hong Kong protest, there was this one moment.

And remember this, I was sitting in an airport. I was waiting to go somewhere and I was on Twitter. And this was right when the Hong Kong police. had shot a woman in the eye. I believe she was a medic or a nurse or something and shot her in the eye and there were these extraordinary photos and the woman on the ground, you know, the photographs all around. It was very, very compelling visuals. And Western Twitter was paying attention all of a sudden, right? It was really paying attention to this moment. My Twitter feed, which is not primarily China Watchers, was all talking about this moment. And it reminded me of Iran.

Actually, when right after the Arab Spring, there was a woman, I think her name was Nihah, who was murdered and her face became an icon, right? And similarly with China, this woman who was shot became an icon. And the protesters began coming out in solidarity with eye patches and things covering their eyes for their sort of woman who became an icon. And when you have that moment where the protest has a human face, has an icon, right? That's where you see the government oftentimes will come in and will realize that this is now a thing that it has to respond to.

This has just gotten quite big. right now there's a face. There's a humanization. It's not abstract protest. It's quite personalized. And so what we started to see almost immediately after that incident happened was all of these accounts coming out of the woodwork to talk about how the West had it wrong. Her own side had shot her. She was really a plant. This was a false flag. The Hong Kong police hadn't done it. When that happens, you see this barrage of conflicting narratives. They don't even have to make sense. They don't have to be cohesive. They just have to be in the hashtag so that when people are searching for information about this protester or about this moment, what they see is this content, which is designed to cast doubt on what actually happened by flooding the zone with alternative explanations. You saw Russia do this when Malaysia Airlines, when the flight came down also, you know, six or seven different explanations for what had happened but how it wasn't them.

when Jamal Khashoggi was killed, the Saudi bots turned on. Boom, here are all the different reasons why Jamal Khashoggi was not murdered, was just missing, was still in the embassy, had fled. So this is a journalist that was murdered in the Saudi consulate or embassy, and just they cut them into little pieces and they got rid of it. It is horrifying. So what they're doing is they're competing for attention, right? So, and they realize, I guess, that they don't need to get a critical mass of all Americans

to do anything. They can just focus on these pockets. diseases can take root literally or figuratively. Swing states can be swung, right? And you focus on these little bubbles, just like in real life, that make fringe perspective seem like a normal or prevailing one.

So you have like antivax, 9-11 truthers, birthers, Pizza Gate, flat earth. And in this case, like you say, you flood the zone with something that people just go, well, we can never be sure what the real answer is because these guys over here are saying that MH7 did this Malaysian Airlines flight crash.

They're saying it was shut down by Ukraine. they're saying it was shot down by Russia. These people are saying it was shot down by the U.S. These other people say it was a mechanical thing. We just don't know. Even though it's like 99% of all the evidence points to one thing, if enough people scream that it's not that and that it's something else, then it casts doubt on what the real explanation might be. And it doesn't even matter. So what's interesting is a lot of people, and I originally thought this too, they're just going to argue their perspective. Hey, this was shot down by Russian militants in Ukraine. No, it wasn't. it was shot down by this or it was mechanical failure. But if they say like 10 different things,

it's even more effective than just persuasively arguing their one counter argument. If you just cast doubt, you don't even have to persuade people, right? It's easier to just create FUD, fear, uncertainty, and doubt. Yep. And so I think one of the things, when we see the Chinese activity, it may not be designed to be persuasive. Right. And they may not care if they lose those accounts because it's really easy to spin up another cluster of accounts when you need them or to go buy a cluster of accounts off the black market if you need to. And so there's a difference in strategy. Russia ran this multi-year long game. And it takes a lot of time and investment in those personas in that operation. And then, of course, when they were discovered and they lost about 3,800 Twitter accounts and a little over kind of

200 in aggregate Facebook and Instagram pages, when you lose that audience, you have to decide whether you're going to go reinvest and growing it again over a, again, a multi-year period. One of the things that we're seeing is the idea that maybe, particularly for American politics, you have enough people who already have that point of view. You can just go in and amplify it. Why create your own accounts? So one of the things that we're looking for for the 2020 election is not so much that same, you know, replicating the tactics of 2016 or replicating this multi-year long-game propaganda operation by this influence operation, but perhaps instead taking those highly polarized existing factions that exist from, you know, real Americans and just,

amplifying the contents that you create that, you know, what could possibly have happened in this moment on this night? Well, here's 10 different real Americans who have 10 different points to do about that. We're just going to amplify, you know, those different perspectives. We've been seeing the state media take that route, right? RT speaks very differently about Russia today, or is it just RT now? Well, it was Russia today, but now the proper name is RT. Okay. RT speaks very differently about the Floyd protests, for example, than its subsidiary redfish does, right? Redfish really leans into the American left,

amplifying content from the left-leaning communities. R.T. takes the more law and order approach, you know, articulating a conservative point of view. And so here, even in a completely attributed way, if you're looking at Russian state media narratives, on one side, they're talking about how

America is burning. On the other side, they're talking about how cops are destroying communities. And so these are, again, these are both points of view held by real Americans. So they're just kind of putting it out there and then they're using their broadcast tools and their ability to reach people to kind of put out these kind of conflicting opposite sides of the story and then retweeting and finding prominent accounts that hold these particular points of view and amplifying them

and trying to get those ideas more exposure. Have there been leaks from the IRA? Like, have we heard from people that have worked there that are like, yep, this is totally real because other people are like, oh, it's not even a thing. That's just an excuse that we're being told. Of course, it's just more fud, right? fear uncertainty and doubt. But have we heard from anybody that's worked though? It's like, no, my job was to cause crap on Twitter. Yes. Actually, Russian press did some of the early

high-quality kind of exposés. Didn't see that coming. Yeah, no, there's some, did it RBC or RBC? I'm trying to think of how they transliterate the name. So there was actually a woman who went under cover. Her name is Ludmila Savichuk. And I probably just butcher the pronunciation of that. but she went and worked as an IRA troll and then kind of came out and told stories about it. And a few others have also told stories about their time there. They had these people who worked in the troll factory and then told their stories. And they described something that sounds very much like a social media marketing agency, right? We have quotas.

We had stand-up, during stand-up. These were the kinds of guidance that we were given for what kind of posts we should put out. The Twitter people operated a little bit differently than the Instagram people. You know, they had to understand the nuances of the platform and react to what was kind of dominant and trending on the platform that day. They talk about sitting next to the person, you know, I'm running the Black Lives Matter page and they're running the Confederacy page and sort of, you know, how are they going to do things that would kind of start fights and increase tension? So, you know, the troll next to you is tasked with starting the other community. One thing that I think is very different about the IRA that gets lost sometimes in the conversation about the memes, though, is that the IRA actually. actively engaged with real activists also.

So one of the things that we talk about when we think about like Cold War active measures is the idea of agents of influence, right? People who work for another government that are not telling you who they work for, but trying to influence you to take a particular action. Right. Like that show the Americans, right? Yes, the Americans is a good. I love that show.

I love that show. Yeah, of course you do. Yeah, that makes sense. And they had like the South African people who were like anti-apartheid or something like that, but they were just being used by the Russian government to stir stuff. And they were just like college students. And one of them ended up, I think, spoiler alert, if you haven't seen the whole series, they end up like killing him for some reason or he ends up, like, getting left behind in some operation.

So it's like they're using these Americans and people who mean well just as tools to further their own ends. It has nothing to do with the actual mission. Right. And so they find people who are ideologically aligned with what they're pretending to be. And social media makes this easy, right? You don't have to have a sophisticated, you know, handsome spy and alias the way, you know, Philip Jennings is the character in that show. You're an avatar on a Facebook profile, right? But at the same time, you're the admin of this page that I follow, and I'm a Black Lives Matter activist, and you look like you're the person running a Black Lives Matter page, and you DM me.

We start having a conversation. You say, hey, can I give you some resources for a protest? Do you guys need some money? We can't be there in person because we live too far away, but would some posters help? Right. And so you start to see this process by which there's extensive amounts of direct messaging that's happening, right? They're really communicating with the activists. They're working to, you know, useful idiot is the kind of like unflattering term to create people who are unwittingly doing what they want them to do because of this perception of shared camaraderie. You know, they do that targeting a range of communities. They hire a Hillary Clinton impersonator. They pay for a Hillary Clinton impersonator to show up to a Trump rally, sitting on a flatbed truck, like in jail, you know? Cute. They hire a self-defense teacher to teach the black community

ostensibly how to defend themselves at protests, right? So again, these are these things that were there, and they're actually paying the guy, and they're paying him via PayPal. They're communicating with them via telephone and text. And he doesn't know that this is all just an influence operation. He's just like, great, a gig that I can run. Yeah, okay.

Amazing. Most people don't think that, you know, the person who reaches out to them on Facebook is a Russian troll. That's not where people's minds go, right? No. Ironically, most of the people that probably believe in a lot of these conspiracy theories would be the people that would be suspicious enough to believe that, except in this case, since the crazies agree with the trolls,

they really won't know because they think they're all on the same side. I'm sure you heard about this. There's like a Texas secession movement and like a pro-Muslim protest. Can you take us through that? I don't remember the exact details, but this was like a classic, ridiculous example of what happens. Yeah, so they had a page called United Muslims, and they had another page called Heart of Texas.

Heart of Texas was the Texas secessionist page. United Muslims was the pro-Muslim page, which they used in some very interesting ways. They used it when they wanted to, they ran like a Muslims for Hillary protest where they wanted people to show up and be proud Muslims for Hillary, which of course was designed to antagonize the other side.

So it was sort of a very, like, kind of dirty way to use that group. But what they did in this particular case is there was an Islamic Center in I think Houston, Texas, Dala Islamic Center, if I recall the name. And what they did was they had the Heart of Texas page come out to protest the Islamization of Texas was what the Facebook event was called. The United Muslims were called to rally on the same day to defend. I'm trying to remember what they said, the specific wording. It was to put out a kind of pro-Muslim presence to counter this anti-Muslim rally. And so people sitting in St. Petersburg created two Facebook events and solicited the members of this group, which were, I think, both were over 100,000 members.

These were not small. Oh, wow. These were not small pages, no. So they had these different pages, and they put out these events, and then people RSD, and then they showed up in person. And so you had two different groups on opposite sides of the street at the same time. And, of course, the organizers aren't there, you know. Yeah, they live in Moscow or St. Petersburg, right?

The organizer, I wonder. Have they ever done anything where there's a protest, and, like, seven people go, like, hey, where's the guy that set this whole thing up? Like, no one's here. Or like, hey, the other side didn't show up. What they kept doing, and what you see constantly over and over and over again in the memes is they're trying to recruit people

to be their local on the ground representative. So like, hey, will you photograph, get involved in the cause, come photograph our protests, come be a reporter for our fake newspaper. So they're constantly trying to find real Americans so that they do have someone to send in person to these events. So that's, of course, part of the operation.

What winds up happening with this Islamic Center rally is the police have to come and kind of like keep the peace between the two sides. And then if you go on YouTube, I actually went to YouTube because I was curious that there was first person video footage. Houston Chronicle covered this, you know, major Texas press is out there reporting on this protest. And you can go back and actually read the coverage from the day, which I think now has a disclaimer over at the top linking to their more recent coverage talking about how this was executed by Russia. But you have this YouTube footage. You actually can go and see a man on the street, you know, a woman with a camera phone recording this stuff. So it actually happened.

And there's, when people uploaded the footage of it to YouTube, which then, of course, provides the trolls with more fodder because now they have video. Right. And of course, this has other real world impacts in the UK. I don't know if this has happened here, but people are burning down 5G towers because they're like, this causes coronavirus.

The 5G towers, so they're destroying and burning these things down. And then, of course, probably the most common, easy to point to example of disinformation in the USA costing people's lives is a lot of the anti-mask stuff is normal Americans, but a lot of it is definitely foreign interference. I mean, you can just sort of tell, as all the hallmarks of it, have you looked at that at all? It seems like that's an easy one for them to get involved with. So one thing I'll say, we have a really hard time with attribution. So the internet research agency, when the platforms provided the data to the Senate, the platforms did the attribution. So I used to get like shit from people on the internet like, well, she said it was a Russian troll, but how does she know? No, the platforms did digital forensics, looked at things like metadata, logins, shared email addresses, cookies, you know, certain types of behavior.

Did this attribution trace the network? And one thing that we found when we looked at the data related to how that the attribution methodology, Twitter provides. some of the metadata, you know, the Baltimore City News fake Twitter account was registered to a Russian B-line phone, right? You had American pages that were being run with, you know, people logging in from Jakarta. And so there was not very much of an effort to conceal the operation because nobody was looking. They paid for the ads in rubles, you know, so this was not a, yeah, yes. Oh, wow. Wow. That was it, they're like, I'm not even going to try and cover this. No, it was just right out there, you know, Beeline is a Russian carrier, sort of looking at a little device data and we're like, Jesus Christ.

Yeah, like, oh, my VPN's not working. Don't even turn it on. No one's even looking. Don't even worry about it. Basically. But that's changed now, right? In any kind of cybersecurity or, you know, kind of conflict situation, when you expose the tactic of the adversary, the adversary should evolve to no longer use that tactic. Right. And so what we have now is an international challenge where there are in tech. teams looking. There are researchers looking. Regular people are cognizant of the fact that Russian bots or Chinese bots or Saudi bots, you know, all any nation's bots exist on Twitter. And so what you have instead is it's much more difficult to say this is being amplified by the Russians. And one of the reasons why we try to hold off from doing that is one, you don't want to delegitimize existing movements. And that's partly not to de-legitimize them, but it's also partly not to absolve them of their responsibility. if it is kind of domestic actors that are pushing out lies or misleading video footage or whatever else.

So with the coronavirus, one of the things that you look at in the work we've done on at Stanford is we've looked at how different countries all around the world. We looked at Russia, China, Saudi Arabia, Iran, Venezuela, Brazil, how each of these governments has responded to coronavirus, both domestically and internationally, ways that they've used their state media apparatus and then the ones that have gone one step further and used this kind of co-eastern. overt, you know, Twitter bots, fake accounts, you know, fake news, that's sort of darker side of the propaganda spectrum. And what we see is there's messaging that's put out for a domestic audience, then there's messaging that's put out for an international audience. And the international content is often designed to, you know, it's just kind of like a public diplomacy effort. We want to look good. So here we are talking about what we've done. But what really matters is, particularly for more authoritarian regimes that didn't handle it so well, ensuring that their population

believes that they handled it well to preserve their ability to govern. And so what you see is conspiracy theory is that the state puts out to ensure that in large part for the domestic audience, because they want the domestic audience to believe this was a matter outside of our control. Dark forces beyond our borders brought this virus here and this is what we have done to control it. But really, the coronavirus was created by the Americans in a lab. where Dietrich was kind of where, you know, some of these sorts of things went. Right. And then you have entities like Russia today which say like the Iranians are saying that the Americans. So it's this kind of daisy chain of just asking questions. And it's been really

fascinating to watch, you know, the thing with coronavirus is the entire world is paying attention to it. Every government has had to, you know, handle the crisis in its own way. The entirety of the population of the planet pretty much has all been searching for information about the same thing as various countries have gone through phases of high death rates or lockdowns. So it's really been a remarkable opportunity to study how narratives move and how this interplay between the kind of traditional like sort of fever swamps of the internet being legitimized through state media and state influencers in this kind of free-for-all. So in some ways, it's been less of our Russian trolls in there and more of a in this sort of

free-for-all in which every faction is participating simultaneously, what are the narratives that are spreading, what are the mechanisms by which they move through communities, and what is the impact that they have? Do we start to see people change their behavior in response to allegations of a drug working or not working, allegations of masks working or not working? How does that become, you know, incorporated into people's identities, their partisan identity in particular in the U.S.? And that's the kind of stuff we've been looking at? Do we know if the United States is doing this as well to other countries? Like, are we doing this in, gosh, I don't even know, like Central Asia, South America. It seems like we would have done.

But I don't know. Are we doing mostly like, hey, we're trying to form a democracy and we're promoting democracy? Or is it kind of like, you know what, screw up. We're doing the exact same thing. We have not found an operation attributable to the United States. We were chatting earlier, And I said there are certain laws that prevent the United States from putting out particular types of propaganda or communications even on social platforms because of the risk of it being seen by American persons and then, you know, violating certain laws. Similarly, doing kind of mass ingestion of social media data to do analysis and find influencers or messaging. That's also not legal. Like there's certain privacy laws that prevent the U.S. government from doing that. So in some ways, these sort of laws that were put in place for kind of older communication ecosystems have limited is the sense that I've gotten U.S. government's ability to do this.

So even though everybody says, you know, you're just not looking for, you're not talking about what the U.S. is doing. We don't have any activity attributed to the U.S. And that is not because people are afraid to find it or unwilling to look for it. It's just that's the kind of current state right now. So the platform integrity teams have been pretty clear that, you know, they will take down any content that they see as inauthentic. And that's kind of where we are right now, the assumption being that domestic groups do come down periodically.

Activist groups occasionally come down. You know, there's a variety of reasons. Sometimes the platforms will pull things down because of coordinated and authentic behavior is the term. That really usually means that there's like a state actor involved. And so the platforms will then, Twitter will make the data set public. Facebook will write a blog post and outside researchers

will do an analysis and publicize their findings. We do that sometimes. We're one of the teams that does those analyses. But the other way that things come down is like for spam, just kind of coordinated distribution, like no allegations that any foreign actor or government is involved. But just like I have a blog and then I have 10 other blogs whose job it is to promote that one blog

and so it becomes more of like a spam type take down. Yeah, that makes sense. The coordinated inauthentic behavior has got to be tough to catch. That's a whole technological discussion. I wonder what other solutions are available to us for this, because especially in this, like, I think Sam Harris calls it an epistemo, I'm going to try and say it. I had it the other day, epistemological free-for-all where facts do not exist. And I've always wanted to use the word epistemological, because it sounds hellosophisticated. But what solutions

are available to us for this if people kind of don't believe any facts now, or if that's like the idea that they're trying to generate or the mindset they're trying to generate? I mean, for me, what I think, we have to fundamentally rethink curation and recommendations online. And I think that that is a bigger issue than state actors or not state actors. I think that's a question of information integrity. And then that's a question of this is where people will yell on the internet and say, like, who decides? Well, somebody's already deciding. The algorithm is already structured in a certain way. So I don't understand why we've decided that these algorithms that are like 10 years old or sacrosanct, and we can't rethink it in any way. So I do believe, and as you started to see with anti-vaccine content

in particular, with Q&On now as well, the platforms are saying we're going to remove it from the recommendation engine. It stays up on the platform, meaning if you want to go find it, you can go find it, but we're not going to return it in search results. We're not going to return it. We're not going to promote it in their recommendation engine. And if somebody is looking for an answer to a topic, we're not going to serve up garbage in response to that inquiry. It's weird to me that in the, you know, span of like five short years or so we've gotten, you know, this idea that, you know, who can possibly decide how to set the rankings and weightings? There is a system that's in place that's that's only that old. I don't think it's unreasonable to think that we could change it. Google in

particular recognized pretty early on that serving garbage in response to health and financial inquiries for search results had potentially very serious negative downstream impact for people's lives. And so it came up with a framework called Your Money or Your Life. This was back in, I think, 2013 that the framework was introduced, 2015, it was improved. It didn't apply to YouTube at the time, interestingly, because for a long time, the search function was seen as kind of different than the social function in the sense that when you're going to search, you have a question. If you have cancer, you want to know what's going to happen to you. You want reliable medical information, not juice fast bullshit. But if you're on YouTube, you maybe want to be entertained, right?

you know, none of these platforms were not the social platforms. They were not developed to be information libraries and repositories of human knowledge where people go find answers to their financial and health and political questions on those platforms. They were to help you like, find your friends, right? Right. Find your knitting club. Not your, not your scientific and medical authority for how you're going to treat a disease or, you know, operate in the world of a global pandemic, right? And as they've been put into that position, as this is what they have inadvertently evolved into, I think it's eminently reasonable to say we have to be rethinking what we're curating and what we're recommending in the context of looking at the potential downstream

harms. On the subject of the foreign actors, though, that's something where, you know, for the 2020 election, we've really done a whole lot of work outside researchers like, you know, our team at Stanford or governments, tech companies, the integrity teams at tech companies, policy teams, civil society, which, you know, particularly for like the African American community that's been very targeted. They have a real strong voice now saying like, hey, come on, you know, these fake trolls pretending to be black people are really detrimental to our community. What are you doing about them? The ways in which academic researchers, all of the different kind of stakeholders that pay attention to and study this topic, now are working together in concert saying,

okay, the 2020 election is coming up. How do we have a narrowly tailored, and nobody wants to be the fact-checking police, but how do we have a narrowly tailored kind of multi-stakeholder ability to say, here's voter suppression narratives, here's voting misinformation related to voting itself? And how do we try to ensure integrity in the election, ensure integrity in the outcome of the election, make sure that people's degree of trust in the outcome of the election remains high, the election remains legitimate. How do we all take the tools and capabilities that we have and work together? So if we see something that looks like anomalous activity, we're not going to be the ones to make the determination about whether it's Russian, Chinese, or domestics, but if we were to

give it to the social platform and say, hey, you guys should take a look at this hashtag, you should take a look at this page, take a look at this account, they're the ones who then have the capability to make that judgment behind the scenes. Or similarly, when someone like the FBI says, this site appears to have financial ties to Russian activities, Russian intelligence services. Right. They're buying ads and rubles. Right. So this is where you start to see that, you know, we all have different capabilities. How do we ensure that we're sharing information in an effective way, particularly as the 2020 election approaches? If we can so doubt and be negative with these narratives, can we do positive things with it as well? Like, I'm kind of thinking of

explosives, right? They can be used for construction, but only like a small part of that would be construction, you can blast the rocks and then you can build the tunnel and that part takes a really long time. But then when it comes to destruction, they're really fast. You don't worry about cleaning up the mess if you blow up like a shopping mall or something, right? You just worry about the destruction part. Can we also do positive things with these techniques and these narratives or these narrative building techniques and technologies? It seems like there should be a way for us to go, hey, remember when everybody started to believe that COVID-19 was caused by 5G? Why don't we use that to show people that gosh, I don't know, good hygiene or like staying in school is a good thing, you know, can we,

is anybody thinking about that or is it just like, hey, man, one problem at a time? No, people are thinking about that. On the immunization front in particular, you are seeing a number of medical journals, you know, running convenings, kind of immunization research teams working alongside marketing teams, not because they're interested in the marketing piece, but again, this idea of everything is a marketing campaign for an idea now. So how do you ensure that you are communicating in a way that's conducive to the modern environment. This is one of the things that's been very frustrating to me as the kind of COVID stuff began happening. In January, we started seeing the anti-vaccine narrative. So we were paying

attention to coronavirus pretty early on because we pay attention to internet conversations globally, and this was a big topic of conversation in Southeast Asia, even in January. And what we started to see among Americans was the anti-vaccine activists saying, oh my God, they're going to use this as a way to institute programs of mandatory vaccinations. So that was the sort of thing where we're like, okay, this is the thing that if there is a vaccine, like the people who are in the government and the manufacturers and the scientists and the public health authorities have to do better than a press conference here and there and like a PDF because the problem that you're getting at is that there are low levels of trust

in authority and there's low levels of trust between communities. And that is not a social media problem. that is a societal problem. And social media can exacerbate it, social media can reflect it. But ultimately, the question becomes how do Americans respond to their authority figures and do they trust them? And if the president that we have, interestingly, funny enough, he was kind of a darling of the anti-vaccine movement when he was running in 2016.

He's tweeted in the past, you know, kind of tacit support for the idea that vaccines cause autism and a range of these other kind of theories. So they were big supporters of the president then. now as this operation warp speed and this idea that the U.S. government is looking for and hoping to find a coronavirus vaccine, they're not quite sure what to make of that, actually. But interestingly, the people who opposed the president who do generally trust science, don't trust the president. And so you have this interesting dynamic of, you know, would you take a vaccine that Donald Trump's FDA, it says it's safe? And that's one of the narratives that you see,

even from people who are inclined to trust scientists and science in general, believe in vaccination, vaccinate their children, are hesitant because they don't trust this particular administration and this particular administration's commitment to science. So the thing that's interesting about social media is it really kind of tears down the veneer of intrallibility. Right. You see your leaders having. Right. I'm trying to think of like a nice way to say it. There might not be, so just go for it. But it's the idea that, like, you know, you can come away at the sense that, like, nobody's steering the ship, right?

Nobody's in charge. And the other thing is, you know, you see the media get things wrong. There are a lot of stories early on with COVID-19 about, oh, it's just the bad flu, right? No big deal. Right. There are a lot of people who were very angry about that. Then there was the mask guidance issue. CDC and World Health Organization didn't update their guidance until people on social media

who were not even epidemiologists had been screened. screaming for weeks that we needed to be using masks. And then there was a sense of, well, random person on Twitter seems to be right and the authorities are lying to us or wrong. So how can we trust the authorities when they come to us later? So you just have this sense that I think we have to rethink how authorities communicate in the era of social media. And that goes for kind of media as well in the sense of saying, hey, look, we got this wrong.

We didn't have the, you know, or not even. and we got it wrong. With the information at the time, this seemed accurate, new information has since come out that renders that obsolete or incorrect, and here is where we are today, right? And or communicating to people, we are 25% sure of this now. And here's our best guess, 25%, you guys are all on here looking for information. We're going to level with you. This is what we know. This is what we don't know. Here's how we should be thinking about this. Right. It requires a very different style of communication than I think these institutional authorities are used to. And I think that that ultimately is going to have to change in order to restore trust in authority

and media. Almost like thinking in bets, right? Have you ever read that book or heard of that book? Yeah. Yeah, I'm a traitor for a while. Oh, yeah. So, yeah, thinking of bets, right? Like, we're not 100% sure.

And we can admit that because I get why an agency says you have to do it this way because this is what we know right now. Because if they say, well, we're really not sure, a lot of people are going to go, well, I'm going to go with the guy who says that he's sure, even though he just pulled the information completely out of the air and has no facts to back it up or research, it just sounds more persuasive. But the problem is once you have to change your mind, then people go, well, how do I know this isn't one of your things where you're just going to flip-flop on me? But if you say, look, we're 60% sure it's this way. And if we do it this way, it's at least

safer than doing it the other way until we know for sure. A lot of folks like us are going to go, okay, good enough. That sounds like a plan. But I think there's just going to be a lot of people that are used to hearing certainty and only will settle for that. And that's a problem. Social media manipulation is right now. What happens now that deepfakes are coming, they're going to be super easy to make. Deepfakes for people that don't know is when, like, I mean, I can only think of porn examples, which might be a little incriminating, but whatever. It's like they put someone's face on a body of another person and they might even fake the voice

and make it look real. I mean, how do we counteract that? If I see a video of Renee DeRest of being like, hey, all this stuff I just said on the Jordan Harpenter show and Joe Rogan about the IRA was just complete garbage. I just made it up. as being paid by Canadian intelligence. I'm not likely, well, I am now after this, but normally I'm not likely to go,

let me just double check that. I saw in a video, it has to be real, right? Yeah, I've been thinking a lot about that lately. You know, as a society, like, we adapted to Photoshop. True. So I've been curious both about, what are the impact of the technology? What is the societal acceptance

and internalization of the technology? Deepfakes are interesting because for those who don't know, with Photoshop, there's a thing that you can go back to, right? there is a original material. And actually, funny enough, Adobe has actually been really, like, leading the vanguard of this desire to ensure that there is some sort of like cryptographly verified kind of certification that says that this is the image as it was when it came out of the camera and here it still is. Or so you can see the manipulations that have taken place on an image or something. When you have something that's generated by AI, there is nothing to go back to. So the interesting question is for audio or video, do people believe it?

Right now we're still in the uncanny valley. It's not quite right. It's not quite there, particularly video. There are certain detections, like ways that you can observe something that a computer has generated. There's little things. Like, you can see when somebody's heartbeat is pulsing. You know, the color of their skin changes ever so slightly. This is how your iPhone can see red to normal, red to normal, red to normal.

Wow, I did not know that. That's incredible. Yeah, there's ways that you can tell, you know, if you're looking at a video of something that is alive. I'm not an expert in this particular type of detection. But what happens now, though, is we have the ability to generate text and still images and video and audio. And I think we focus a lot on the video because it's very, very sensational. As you mentioned, it also did kind of really take off in porn first. Yeah, that's where most people get their news these days is right out of porn hub.

Like new tech? Oh, let me go look it up right now. Porn started taking it down out of non-consent, actually, out of the idea that the people who were in it had not consented to be in the video. Imagine looking to porn sites for like guidelines on how we're going to develop policy. Yeah, on how to develop policy for it. So that was kind of an interesting thing. Facebook and Twitter have since come down on generative media. They've recognized that it's different than something that's just selectively edited. Selectively edited is allowed to stay up oftentimes with an annotation. Generated will come down if it's seeing as being manipulative. Question is what happens

when you have generative that is labeled a satire. Does it fall under artistic expression? So there's a lot of these kind of policy gray areas. But the thing with the video and audio is that usually they are used to create sensational moments, right? So this is a hot mic politician or a sex tape politician has been, you know, of course, because everything goes back to that. So those are very sensational moments that then inspire people to go and try to authenticate the video, find out who created it, find out how it spread. So there's a lot of attention that kind of immediately goes into understanding that particular video. The thing that's been more interesting to me has actually been like GPT3, which is the open AI's text generation AI. So this is like an AI algorithm or program that writes like a human?

Is that kind of what that is? It writes text. Yeah, you give it a prompt. You can give it kind of constraints around how creative it can be. The AI has been trained on content on the internet. So it's read Wikipedia, right? And so it has a body of knowledge, which is very interesting. And so I've been using it just to see what comes out of it.

depending on what prompt you give it and how, whether you constrain it to kind of stick to what it knows or allow it to be more creative, you will get content back. Very different content. You can submit the same prompt over and over again and it'll return very different types of content to you. You can have it write long form essays. You can give it tweets and it will return back kind of tweet link things. I've been working on an essay actually and I thought, him having a hard time with the closing. It's an article on AI. Let me give it a GTT3 and see what the AI generates for me for my closing. And I did a couple really interesting things. Like once or twice it pulled in characters I had not mentioned. I was writing about AI and all of a sudden it gave me a paragraph on Edward Snowden.

And I thought like, okay. I wasn't expecting that. Once or twice it returned links to like returned regurgitations of academic papers that it had probably kind of read at some point consumed. And then a couple of times it gave me back like once it suggested a title were the, I gave it a couple paragraphs that suggested a title. And then it gave me, you know, how when you go to, like, you read an article and it gives you the list of, like, related articles on the bottom. Oh, yeah.

So it, like, generated a whole bunch of titles for things that it thought would be, like, related to this content that I had given it. Wow. Which I thought was sort of an interesting response to get back. But what the takeaway for me was, like, it's not perfect. You know, it is sloppy. And at times it goes off the rails into these, like, freshman philosophy essays. Yeah, I mean, that's as far as I ever got with it. So it probably just looks like I wrote the essay.

But, yeah. As Emmanuel Kant once said, yeah. Or it is like garbage, just like word soup where you write enough words on a page and you hope that something has come out of it, but really it hasn't. Right. Yeah, you hope your teacher's grading it while watching Netflix. So it's not perfect, but it's remarkably good. And with like some mild human curation, you could see how if you wanted to run a Twitter bot, you could give the prompt to run it a bunch of bunch times, pull the stuff that comes out of it, evaluate it, you know, check it off and push it

into a queue and then go, right? And it's original content. And that's the thing that's so remarkable about this stuff. You can't trace it back to something somebody else said. It's not a copy pasta. It's not an engram that is like, you know, tied to some community or something in the past. It's uniquely generated content. And so it really is interesting. It kind of in a way, you know, democratizes the ability to run a commenter army because all of a sudden you have this machine that does the work of producing all the content that your, you know, kind of 50 cent army had previously been paid to do, right? So you have this ability to create something that's virtually undetectable. It's cheap. It can be put into an environment where it's not

sensational. People aren't going to go look at it. It's just read as like, this is the voice of the man on the street. This is the opinion of this commenter. And so what I think it ultimately does a lot of these things is it means that we have to have a kind of different relationship to our idea of identity online, right? When you can no longer trust that the content you're reading is being produced by a real person, when you can no longer distinguish, then the question becomes like, Lauren, is do we care? Do we evaluate the comment based on the merits? But as we've talked about in this conversation, if you create the perception that a whole lot of people have this opinion and you've just done that, you don't even have to use copy pasta now. You have an AI

generating that, then you really do have the ability to influence through this repetition and this persuasion and this idea that everybody's talking about or thinking the same thing. So then the question becomes, how do we know that these are real people, real identities that are tied to this content? And so I think that the next thing that we'll be thinking about is actually that question, really. It's how are we going to think about identity in an era when faces are generated? You know, when I was, when I spoke with Joe Rogan, and I said there's this website,

this person does not exist.com. And it generates faces. The AI is generating faces, and each face is different. and there's no permanent repository or library. So once the face goes by, if I've screencapped it, nobody's going to find it because it's not a stock photo and it's not a real person. And as that technology improves, again, you can create personas with these computer-generated faces that are untraceable.

And we've already seen three or four influence operations. I was talking about it with Rogan in March of 2018. Sorry, I spoke with him about it in March 2019. Yeah, this year's been five years long. It's fine. Right. So it's really confusing. being that 2020, yeah, has been at least three to five years in my perception.

So I was having this conversation about this technology in March of 2019. And in November 2019, we had the first takedown based on this network of pages that were attributable to an entity affiliated with the Epic Times called the BL. I think it stood for the beautiful life or something. One of these kind of, you know, fake is not the word. One of these like kind of spammy Facebook pages was creating personas to share their articles. and they use your fake accounts with AI generated faces, just going into groups and sharing content affiliated with this publication. So we've already seen ways in which people who want to manipulate are using all of these different technologies as they become available. And so the

question becomes, okay, how do we think about in advance what kinds of frameworks we're going to want to see to ensure that people can trust the information and content that's in front of them trust that that person who's saying it is real. Renee, thank you so much. This has been fascinating. It's enlightening. It's also a little disturbing because it almost seems how optimistic are you that we're going to be able to solve this?

Like what happens in other countries like Estonia that have been hit with Russian influence operations for decades? Like, are they doing okay or is it just nobody trusts anything they see ever? I think there are certain countries that have a higher degree of trust in government than we do right now, right? So like Sweden has been a target for operations, but people trust the government. when the government says, you know, this is how these operations look, people internalize that, and then they feel better educated and the impact is less. That doesn't happen here. Unfortunately,

it turns into, well, this political party said it's a thing and this political party, you know, just is saying it because they hate that other political party. So it's become a partisan debacle here. People have adapted to emergent technologies before. Propaganda's been around for a very long time. I think the question becomes, are we sufficiently prioritizing making changes to, to improve our information environment, right, to recognize that, you know, this is the new normal and how do we want to adapt to that, either by way of regulation or by way of changes, you know, making algorithms more understanding of the downstream impacts that they have. So I think that that's kind of where we are today. I wouldn't say solve. I don't think that you solve this information.

I think it's more like a chronic condition, right? You manage it. You adapt to it. You help the population and be resilient to it. And that's where we have to be going. Renee DeResta, thank you so much. Thank you. I've got some thoughts on this episode. But before I get into that, here's a preview of my episode with Matthew Shrier. He was a freelance journalist kidnapped in Syria by al-Qaeda. Locked up for months at a time in basements all over Syria.

Matthew suffered and learned a lot and eventually managed to escape his captors. This was a crazy conversation. Here's a snippet from that. Boom, this silver jeep Cherokee just cuts across from the oncoming lane and forces us to a stop. The doors popped open and they got out. The guy in the front seat, you know, it's called head to toe in black. He had an AK in his hand, dude in the back seat, just this pock-faced guy sweater with a chrome pistol in his hand. They jumped out and I knew exactly what was going on.

I was just like in shock. Dude in the black came over, opened the cab door, takes me out, leads me up to the Cherokee, puts me in the back seat, he gets in after me. I looked at him. He reaches up. He pulls the ski cap I was wearing because it's cold in Syria in December. this is New Year's Eve. He pulls it over my eyes and leads me forward and presses the barrel of the rifle to my head. And we took off a couple seconds later.

I still didn't know who had me. So, you know, the way to figure out who has me was I asked for a cigarette because, like, pretty much everyone in the Free Syrian Army smokes and anyone in the gang will smoke. And when they told me I can't smoke, that's when I knew I was in really deep trouble with the El Nostra Front, which is al-Qaeda. And they bring me up to hole into the boiler room. And that's where they torture people. There's kids everywhere.

there's a guy hanging from a pipe by handcuffs. They sit me down with my knees bent up to my chin, and they force a car tire around your knees. And they take an iron rod, and they slide it over the tire, but under your knees and the crook, and that locks it into place. And then they flip you over on your stomach,

so you're cuffed, and your feet are in the air, and you can't move them. And they take this thick cable, and that's what they use. They start wailing on the bottoms of your feet. feet. Let me tell you something. It freaking hurts. And I got 115. That was the beginning of our punishment. What are you out of your mind? We're trying to escape from a terrorist prisoner. We have more to worry about to get in our arm jam between a rock and a hard place for 127 hours. He's like, well, I never saw

that movie. And I was just like, oh! To hear how Matthew survived captivity and escaped being held hostage by Al-Qaeda in Syria, check out episode 217 of the Jordan Harbinger show. There was so much. We just ran out of time. I mean, there are fake YouTube channels. There are fake podcasts. The Chinese Communist Party does this too. They call it White Monkey Jobs, which I think is interesting. There was a whole fake YouTube channel called Williams and Kelvin is all just to create fear, uncertainty, and doubt. They were just hired by Russian intelligence operations. They had no idea who they were working for, and they were just getting money supplied by foreign agents. Just crazy. I mean, it makes perfect sense. It's so easy to just go on websites like Fiver or hire

other YouTubers to create video messages that are sponsored and make it political or make it so division. Now, what we can do, Renee was telling me, is we can look at the dates of social media accounts. Is the account new? Does it have 40,000 retweets in the last two weeks? That screams automation. Is it a singular topic focus posing as a real person? You know, somebody who just only talks about anti-vax 24-7 and seems to never go to sleep? Also, the IRA understands the nuances of meme and internet culture, and they're so adept at this communication. For example, they use people of color only in memes on black LGBT pages, for example. They know better than to use white people in the photos. So there's an extreme level of, they're very adroit at what they're doing. They know what they are doing.

They understand the subtle cultural differences that are going to make something stick here in the United States. They just keep testing it. There's hundreds of people working here. This is all they do. Also, the pages, I thought this was interesting. The pages transform from one thing to another. so they might start off as like black gun owners and then it becomes oh it's an anonymous page for the hacker group anonymous they're often sleeper accounts they'll go from the simpsons to bernie sanders to second amendment jesus it's just crazy or they'll start off as something really innocuous like the simpsons and then they build a following and then they overnight get very political and change the subject matter after building a couple hundred thousand followers or something like that the platforms know they are

serving this BS. They just want us to stay on the platform longer so that they can serve us ads. And the recommendation engine on things like YouTube, it automatically does this for this reason. Platforms need to admit this is a problem before we can get our hands on it and actually fix it. So how do we tell if what we're seeing is an influence campaign or if it's quote unquote real? Well, does the article that's being shared support what the headline says? Read things before you share. Think about things before you share. Ask people that you think might know better before you share. A lot of people are. are asking me stuff all the time about this, especially on Twitter. Also, does the post read

like American English? You can also see a lot of bad English in some of these fake and info warfare type posts. Of course, that's not a foolproof way. You can also see things that are repurposed American content. There's plenty of crazies here in this country as well. Is it automated? Is it all the same thing? Is it all memes? Doesn't guarantee that it's a bot or automated but is a pretty good hint. And by the way, we are not the only country facing these sorts of attacks. Countries like Estonia and other nations in the Baltic region and Eastern Europe, they've been dealing with this for decades because of their proximity to Russia and large domestic Russian population. So this isn't a problem that's new and it's not a problem that's going away

anytime soon. The solution is education. We are the ones that have to know how to filter this information. And you know damn well that you're not going to be able to train crazy Uncle Frank. So it's on you. Remember, everyone who believes all this crap, they also go out and vote. And they reproduce for that matter. So make sure that you are doing the same, or at least voting and educating yourself if you don't want kids. Big thank you to Renee DeResta. Links to her stuff will be in the website in the show notes. Please, if you do buy any books from any of the guests we have here, do use the links on our website. It does help support the show. Worksheets for this episode in the show notes, transcripts, in the show notes, there's a video of this interview on our YouTube channel at jordanharbinger.com

slash YouTube. I'm at Jordan Harbinger on both Instagram and Twitter, or you can just hit me on LinkedIn. I'm also teaching you how to connect with great people and manage relationships using systems, using tiny habits so that it's not a huge pain every single day. That's over at Jordan Harbinger.com slash course. Do dig the well before you get thirsty. If you're trying to create relationships once you need them, you're too late. And most of the guests on the show, they're in the course. They help out with the course. Come join us. You'll be in smart company. This show is created in association with podcast one and my amazing team. That's Jen Harbinger, J. Sanderson, Robert Fogart, Ian Baird, Millie Ocampo, Josh Ballard, and Gabriel Mizrahi. Remember, we rise by lifting others.

The fee for this show is that you share it with friends when you find something useful or interesting. You know somebody who's interested in info warfare or Russian interference or just interested in how these platforms work and how people are persuaded. Go ahead and share this episode with him. I do hope you find something great in every episode of the show. So please do share the show with those you care about. In the meantime, do your best to apply what you hear on the show so you can live what you listen, and we'll see you next time. This episode is sponsored in part by Something You Should Know podcast. Finding a new great podcast shouldn't be this hard, so let me save you some time. If you like the Jordan Harbinger show, you'll probably like Something You Should

Know with Mike Carruthers. It's one of those shows that makes you smarter in a practical, useful way. Same curiosity vibe we go for here, just in a fast-focused format. Mike brings on top experts and asks the exact questions that you'd want to ask, and the topics are all over the place in the best way. Recently, they've covered things like why we care so much what other people think, the benefits of laughter, why sports fans get so invested, and what makes people like you or not, the through line is always the same. Smart ideas you can actually use in real life. Something you should know has been featured in Apple's shows we love, and it's got thousands of five-star reviews because it's consistently interesting. So if you want another show that scratches that

I want to understand how people in the world really work itch. Search for something you should know wherever you get your podcasts. Look for the bright yellow light bulb and start listening. You can thank me later.