The Vergecast - The history of online harassment before and after Gamergate with Caroline Sinders

Episode Date: December 4, 2018

Cataloging online harassment before and after Gamergate with Caroline Sinders Learn more about your ad choices. Visit podcastchoices.com/adchoices

Before we begin, here's a word from our sponsor, Dell. Today's segment is sponsored by the Dell XPS13 laptop with an 8th-gen Intel Core I-7 processor. Experience Dell Cinema's incredible color, sound, and streaming on the Dell XPS13. It's the laptop for people who watch things on their laptop. Learn more at Dell.com slash XPS13, sponsored by Dell and Intel. Hey, everybody, it's the night from the Vergecast this week on the interview episode. Casey Newton and I sat down with Caroline Cinder, who is a machine learning expert, a designer, a product designer, and a fellow at the Harvard Kennedy Digital School where she's compiling an exhaustive history of online harassment. Now, most people think harassment began in 2014 with GamerGate, but it started long before that. GamerGate was just an inflection point. We talked about that a lot.

We discussed how other scandals like Donutgate, PizzaGate, how they keep growing, how they've turned into a playbook for other harassment campaigns. online, and we discussed all of the tactics that harassers online use and how the platforms can fight back through smart product design. It was really interesting. It's really important, and there's a lot of work to be done, including on the individual level, work that you can do to make a harassment better. Also, as we were recording this episode, Laura Lumer, who's a bit of a right-wing crazy, had literally chained herself to Twitter's doors in New York. So that was a delightful coincidence. You'll hear us joking about it all throughout. Check it out. Caroline Cinder. All right, we're here with Caroline Cenders, who is a machine learning designer, a researcher, an artist, your fellow at the Kennedy School. What does that mean? So a fellow can be a much of different things. What it means is I am funded by the Kennedy School, which is actually a program called the Digital Harvard Kennedy School. So I'm a fellow exploring different kinds of problems on the internet. And so they're funding like a piece of my research. So I'm working on a very specific kind of project for them. I'm looking at are there,

patterns inside of communication apps and thinking about communication apps broadly. So IMessage, Telegram, Threebo, Sinaweba, Twitter, anything on Facebook, so any surface of Facebook as a communication app. And are there patterns that elicit trust that cause users to trust those things more? So is it rounded corners or bright, happy colors, or things that look a certain way? Is it beyond just users having their social network there or different people that they want to talk to? Are there other things that sort of engender trust on these platforms? And then do those design choices, then off you skate poor security. So that's what I'm studying at the Kennedy School.

That's super interesting. Thank you. And then you also run a firm that does machine learning of some kind? I run a design firm called Convocation Design and Research. One of the things we focus on is machine learning, but it's a design firm. and I like to solve complex problems and make them human readable. So some of that is machine learning. Some of that can be ethical quandary.

Some of that can be trying to take a technical challenge and present better design around it. And I think some of my work from the Kennedy School fits into that. So also thinking about how do you design for NGOs and nonprofits or public good? How do you translate machine learning into different artifacts or entities or writing that works for a more common audience? So that can be, you know, doing high-level consulting and talking, you know, how to explain things in plain language, right? Instead of saying everyday speak, how do you think about translating technical documentation or technical processes in a way that's understandable? Yeah. So you, like, apply the machine learning to that. Like, we're going to read a stack of NGO white papers. No, no, no. So I will take a machine learning process or a product and explain to someone, people don't know what you're doing.

I see. And because I understand how machine learning works. And I've. worked as a product designer in machine learning. This is how you should maybe think about explaining it. Yeah. That's great. So you just put together what you have titled, Incomplete but Growing History of Harassment Campaigns. And it starts in 1990, which is like my favorite thing, like the first IRC fight.

I asked Casey to join me because he obviously covers speech and democracy and platforms all day long. And so I just want to start. Why did you feel like you needed to chart out every major harassment campaign since 1990? I started this actually in 2016 towards 2017 because I realized at that time, and this is like the very beginning of like the alt-rate, really growing online, that so many things were being repeated. And then this is a big thing when we talk about why we teach history, right? Because history is doomed to repeat itself.

What I also found fascinating is when I started researching online harassment in 2013, how much of it was just the same kind of tactics replicating itself. and we saw this in GamerGate, especially. So I thought it would be important to actually sit there and outline everything. And I especially thought about this because when I started my online harassment research, I was really focusing on GamerGate. And that became something I was known for talking about and exploring. And also I explored that from a product design standpoint of how the certain structure of platforms

and the design of platforms can actually help a harassment campaign. What are ways to think about redesigning platforms to mitigate harassment? But there were so much online harassment before GamerGate. And so a few of it was getting covered, especially if you look at your slip is showing, which I argue in a lot of black Twitter feminists argue this as well, or black feminists on Twitter, who used Twitter also as their medium for exploring and documenting different kinds of social media culture, that a lot of the tactics used in your slip is showing. Can you quickly explain what that is?

Okay, so your slip is showing is related to this 4chan harassment campaign called Operations. Lollipop. A fair amount of harassment campaigns will have Operation in the name. Operation Lollipop was planned on Forchan as a harassment campaign to impersonate women of color feminists on Twitter to drive wedges in actual feminist community between white feminists and women of color and then between women of color each other in feminist subcultures and communities. And it was around this hashtag called In Father's Day, which was a fake hashtag, so we could call it a hoax. that would cause a schism, and it did cause a schism. And this was planned.

And these were fake accounts, impersonation accounts, right? And a handful of black Twitter feminists knows this, or black feminists on Twitter. And one in particular is so true. And she created a hashtag called Your Slipis Showing. And that was shorthand to sort of say, you're fake. Yeah. And it was a colloquialism.

She uses hashtag and tweeted at people, Your Slip is showing. She then took a Storify account, and collected all the time she's used it to then document all of the fake accounts. Yeah. What we're seeing here is the impersonation of people. So also, and by impersonation, actually, like, creating fake accounts and fake entities to create noise and to game and flood a hashtag. And then this hashtag was also a hoax.

This is something we saw in GamerGate. This is something we see right now, actually. But it's not around misinformation or political news or around our daily news. it's targeting specific people. It's targeting communities. And I think it was really important. And one of the reasons I published it now is I think if we're starting to look and study misinformation, disinformation. We don't look at the history of online harassment campaigns and social media.

What we're missing is a massive part of this history of seeing how people have used these platforms for many years. Way before we started talking about fake news. Way before we started talking about Gamergate. So that was like the main impetus behind creating this was, well, we're missing actually all these stories. And these stories are important, not just because these victims are important and their stories are important, but this is proof that, like, different groups online have spent years honing how to create misinformation campaigns. It was just under the realm of harassment. It just wasn't under the realm of misinformation or disinformation the way we think about it today in our political culture. And so as you read it that way, do you come away thinking if only we had taken this seriously sooner, maybe we would be, in better shape than we are now? Totally.

But one thing I want to highlight, and that was a great question, thank you for asking, is women of color in particular have been talking about this for years, like social media historians, activists, advocates, writers, Siddette Harry in particular,

who's with Mizzilla, and in her own right is her own researcher outside of her work, outside of her day job. People have been talking about these things for years, and they've been talking about

these as planned hoaxes and as coordinated campaigns. And these are key words, hoaxes, coordination, because we see them right now. They're just not about politics necessarily when these occurred in like 2011, 2012, 2013, 2014. Are these like fractally similar? Like, does the shape of End Father's Day look like the shape of Russian interference? Yes and no. I would say from a qualitative or ethnographic standpoint, sure, like the way in which people coordinate and impersonate, I'm sorry, I keep using impersonate.

I'm sorry, I keep using impersonate, it's the wrong word, but create bot-like accounts. but they're not bots. They're human-powered bots, right? So sock puppets. Yes, thank you. They're creating these sock puppet accounts. So, yes, it is similar in that form. But is the scale of it?

Similar, maybe not. But what's fascinating is scale kind of doesn't matter when you're trying to figure out the edges of where a system breaks. Because if you can do it with like five or 20 accounts, you could probably do it with hundreds or thousands, right? If you can go through and plan and coordinate this kind of thing and you're not tipping anything off.

and you can do it for a while, you can probably scale it, right? And if it breaks at scale, then you dial it back and figure out where it works. And that's what I think has been happening in these years. And this is what we're seeing. So I always think about Gamergate as being this enormous tipping point. We run a tech website. We cover video games. Our sister site is Polygon.

They cover video games. We, like, were in it. We were just fully in it. It was a big deal for us. And so to me, it's, like, very personal. and it just feels like everything changed after that and now you can like, there's a hundred headlines out there that's everything is GamerGate now.

Totally. Is your point kind of like that was just when everyone noticed and everything is, this has been going on? Like, how should I think about that? I would say yes to part of your question that this has been happening before Gamergate and we should have noticed. But GamerGate was a tipping point.

So I also started researching this because I was making video games. And I saw so much of like my own community targeted I was making video games and also getting my master's at the interactive telecommunications program, ITP, where I was working and studying under Clay Shirky, and we were looking at social movements or political movements on social media. And so Gamergate, to me, had all the clear hallmarks of a political movement. It was just a harassment campaign. But it was big as a political movement. I think that is also what makes it different.

It wasn't a one-time thing. It wasn't just for the lulls. And I mean that in very particular way. If we look at anonymous campaigns, for example, or we look at anonymous campaigns, for example, or we look at the history of 4chan or B, doing things for the lulls is very short, and it's supposed to be short. It's supposed to be something you do

as a joke, and then it's done, and then you do something else to one-up something. Gamergate was the sustaining of a movement. Using tactics, though, that we have seen from before Anonymous became, like, White Knights and Black Hats, for example,

right? Or before Anonymous had a much more political bent to it, when Anonymous was really just anonymous on Forchan. Gamergate took it almost a step further because it was a political movement. It also was an emotional movement for those that supported Gamergate. And this is not to diminish the harassment that they did. What they did was extremely violent and life ruining for a lot of their victims and very serious. But I think for a lot of those that were supporting Gamergate, it was also, it was a political movement.

It was an emotional movement because of something affecting their fandom and something they had built, like, personal parts of their lives around. and it wasn't about a one-time thing. It wasn't actually about irony or humor in the way we think about, or in the way, rather, you can contextualize 4chan movements. It was something different. And I think that's also one of the reasons it gets talked about because if you remember back to when Gamergate was first covered, people were really dismissive of it.

Oh, it's just infighting in games. Oh, ha-ha. Like gamers are just like virgins and their mom's basements covered in Cheeto dust. And that is like, the way people describe trolls, which is not true, right? And then when it kept going and kept going, then I think a lot of the general media understood that it was something different. And I think that's the reason it was this tipping point.

It happened to be this collection of just the way you describe it, inviting in games. That already sounds kind of strange. Yeah. Right. And then death threats, swattings, people quitting their jobs, bomb threats going into workplaces, right? are into conferences.

And then you see that it didn't stop after a month, and then it kept going. And then if you actually looked at who it was targeting and where it came from, it just kept getting more and more extreme and also more and more strange in the way in which online campaign would go. There was not a lot of things that looked like it in terms of length of time, amount of victims, and the severity of it. What it is is it looks like a protest campaign. It has probably the size of something that is a much bigger political movement, except that this was a harassment campaign.

So your piece, what is great, everyone should check it out on Medium, currently has over three dozen harassment campaigns. And when you talk about GamerGate being a tipping point, I can't help but notice reading your report that it seems like the pace and the scale of these harassment attacks are accelerating. Is that fair to say? I'm going through the report with a data visualizer to start tagging them and trying to look at the size.

So citation needed, but that seems like a correct assessment. And, you know, going off, like, let's make this a hypothesis that can be tested then. So if they are accelerating why, a potential answer could be because maybe the same people were involved if they're on the same platforms and they're learning each round. And that's something that I interviewed over 40 victims of GamerGame. And one thing someone said is it felt like, because they had been a victim of your slip is showing that it felt like people, the same people were involved in GamerGate.

Yeah. And, you know, that needs to be verified in a way. But what's important maybe is the fact that someone felt that because what that means is that they're picking up on similarities that exist. And I think we can look at these and say that there are some similarities. Sockpuppeting, amplifying a hashtag, dog piling on different victims, doxing them, swatting them, death threats, rape threats. Maybe those are also just regular, you know, more reoccurring forms of harassment if we

think about it. But if those are all the things being used and they're not being stopped, then maybe they are getting bigger and faster every time because there's no friction to stop them. Right. Also, like up until a certain point, the internet was growing, right? Like in 1990, you just don't have as many people using online communication tools as you do by 2018. So I think that has to explain some portion of it. But reading through particularly the most recent examples in your report, which would include like donut gate, Pizza Gate. It's hard not to look and see a lot of tactics being repeated. And I think to some extent, this does feel like a playbook that people are just running whenever they encounter something they dislike.

I think that that's a pretty valid assessment. I mean, Gamergate, for example, wrote out when they were starting a campaign, a micro campaign inside of Gamergate. So if they were deciding to attack the advertising of different media websites. They had a list of what you could do and how to do it, and they would publish this on Kataku in Action on Reddit. That's their subreddit. So organizing is a major thing, right? And I think B. L. L. Coleman's book, Hex or Hoaxer Whistleblower Spy, gets into this a little bit with Anonymous of how they would also create a campaign and then galvanize their followers and also disseminate information. So I haven't seen, per se, one playbook passed around from Gamergate to the alt-right, but also you kind of don't need that playbook

when you can see the effects of what happened in a previous campaign. Right. You can reverse engineer these tactics pretty simply because they're not sophisticated tactics necessarily, right? They're not. There's like so many things you can do. It's just all the things you can do are extremely detrimental and dangerous, right? And very harmful.

So you talked about platform friction. Literally, as we're talking today, Laura Lumer is chained to the front door of Twitter in New York. And it's insanity. That's just, 2018 continues to happen, like at a furious pace right up until the end.

But what I always laugh about is like, I say this out loud. I don't even know if I fully believe it, but like Twitter is not the internet. Right? Like there's a great big internet out there. But Twitter seems to be where this stuff happens. It's the epicenter of all of these things constantly.

Is it up to Twitter to provide friction? Is it a collective action problem? Does it actually happen on other platforms and we just don't talk about it as much? So there's two answers here, and I want to break them apart into two different answers. One, I think we just stop thinking about Twitter as, like, a major part of the Internet when it's a major part of the Western English-speaking world. There's so many other forms of the Internet out there, right? Like, what's happening on Sinaweba right now?

That's a major, major part of the Internet. What happens in mainland China on the Internet is major? What are the different kinds of social networks in Russia? What are they talking about? How do people interact on there? Right? What are the platforms of choice inside the Middle East? How are people communicating?

You know, our internet is incredibly partitioned because of capitalism and because of major technology companies. And I think that's really important to highlight. Like, when we talk about the internet, whose internet are we talking about, first of all? And what languages are read on that internet? or are more commonly used. So the internet is different now than it was in 1990. But the internet in 1990, if it was seen as a utopia,

it was also a white Western utopia. Who got to access the internet in 1990? How easy was it? How affordable was the hardware? Where could you purchase the hardware? How could you connect to the internet? Did you have the bandwidth to connect, right? So all these are actually very important design choices to consider.

This is a designed space. An internet, a space on the internet is not just an open space like going into a park, because someone had to write code to render that space that you're in. So that means that a choice went into that, and that, I argue, is design. Pushing that aside, though, for us in this room, people based in the U.S., probably a massive part of our internet based off our ages, is probably a social network. So what are the social networks we use in this room? Reddit is a social network, Facebook, Instagram, Twitter. These are social networks of choice.

What is different in a way, I think, than Facebook, which is a walled garden, is that Twitter has the illusion of openness. And it's much easier to search. You don't have different privacy settings. So you can't be semi-public. You're either public or private. So when we think about what are frictions that could be added, there's lots of things. things social networks can do. When those things happen, it is less obvious to like the common public. Do you see all of the harassment that someone faces on Facebook? No. And some of that is

by design because that does also make it easier to mitigate some forms of harassment, right? But just because we don't see it, that doesn't mean that it's not there. And that doesn't mean that it's not viral in a way. So I think it's important to unpack how we're contextualizing things like harassment. Is it on a platform where virality? can happen? Is it on a platform where media is easily embedded and shared? Is it on a platform where you can take something down? Is it on a platform where an algorithmic timeline can quickly intervene to where you see something later? So it's not happening in real time. So you have delayed effect of interacting with information. All these are our design choices. All these are

important choices. So going back to your questions or something to where you can do, there are plenty of things they can do. I've just outlined some of these design choices. Should they do them, It's a different question. And how they do them is another question. How they do them is the thing that I worry about the most of Twitter, right? Like one of my recurring themes in my work is that Twitter is not a competent company. And I'm not trying to say that in a mean way. I'm trying to say like they try to do things and the implementation often just goes wonky, right? It is at a service where things just break because they're held together with spit and glue. And so I will give them credit for at least beginning to take these issues more seriously. They've done an RFP. They're working with academics. They're trying to understand what it would mean to have a healthy platform. But they are just in like the research stage of that, right?

And so how you get them from there to implementing a policy that meaningfully restricts harassment, that's just a very long way for them to go. I want them to get there. It's just at the moment I'm not sure how to do it. They've been researching and tackling harassment since 2014. Like they had an advisory council that they, brought a lot of different activists, academics, and policymakers to. They had women action media conduct what at the time was like a canonical survey for them on online harassment.

And they had worked with different academics. So I don't give Twitter a pat on the back because they've had time since 2014. They've spoken to very well-known victims of Gamergate multiple times. Jack, apparently if you're famous, is very easy to get hold of. And we have so many cases of harassment, right? We can look at Ghostbusters and Milo Yonoplas. I forget the exact actress's name. Leslie Jones.

Yeah, Jack intervened in that. We can look at me too and what happened accidentally with Rose McGowan's account. And Jack intervened in that pretty quickly. So I think it's disingenuous to say that they're only trying now. They've been trying for a while. I think a better question is, what are you doing and why are you moving so slow. And if that's because they don't know how to do something, they're not transparent about that at all. I think Twitter is one of the most, most opaque companies. You don't know why they

make a decision. They don't publicly release a lot of things. Facebook actually looks more transparent at this point than Twitter does. Are we liking the outcomes of what they do that's different? But like, is there more insight into their processes? There is. Yeah, they could say a lot more than they do about the decisions they make and why, and they don't seem to be there yet. So I want to ask you, I want to bring this all the way back to the thing that you said you do at the top, which is you help government organizations NGOs like communicate more effectively through design. So Mark Zuckerberg is out there saying, I want a Facebook court. Every time he says this, my little lawyer brain explodes.

Because that implies we're going to have decisions. There's going to be lawyers. You're going to issue precedent. We can then argue about whether the facts of our current case fit the precedent of your previous. decision. Like, that's a huge investment. Totally. Most countries can't do that well. Yep.

But how do you make that understand? Is that even the correct outcome? Can you make that understandable to people? So there is actually a history of this kind of committee in social networks. If you think of Wikipedia as a social network, there's the arbitration committee. So prior to joining the Kennedy School, I was actually on the anti-harassment team at the Wikimedia Foundation. The arbitration committee is a group of administrators.

So it's regular Wikipedians. elected to an administrative level called ministers. And then they run for the arbitration committee, which you can view as the Supreme Court of Wikipedia. And they hear really difficult cases that have not been able to be resolved. Wikipedia doesn't have a way to report harassment. It doesn't exist. They have different places where you can try to talk about problems that you're having or harassment that you're facing inside of the encyclopedia. And administrators can advise or try to do things. And if it keeps getting out of hand, you can try to file a case to the arbitration committee.

Now, those on the arbitration committee have shown that they have contributed to the community in some way. What's important here and what is criticism of the movement is that doesn't include that you have a background per se in mitigation or litigation or research. What it's shown is that you're a community member withstanding. What is a community member withstanding? What does that mean? How does that scale? Does that mean you have a background in understanding diversity? Does that mean you have a background in handling certain cases? How, what's considered the way in which cases are handled well? Well, you look at the cultural norms of the community. So what are the cultural norms of the community? Wikimedia projects actually have a history of

toxicity. They're very toxic communities and it's pretty well reported. And it's well known inside the communities itself. So what are the norms here? What are the things that are sort of ruling in a way that are considered normal for this space? Well, it has to follow the norms of the space that you're in. So this often defaults to preserving the encyclopedia. It often goes back to looking at how someone editing. It often looks at their standing as an editor.

Are all these things that should be considered if you're being harassed or not? It gets really complicated. So it's funny that you're saying it gets really complicated because you're describing this to me. And this is every criticism of the jury. racist to me in the United States. Totally. And I think, and the thing about it is, and this is where perhaps I differ as a member of the Wikimedia world, is I don't think that it necessarily works very well. Yeah. It's also the foundation doesn't intervene in these decisions. There is a separation between

the arbitration committee and the Wikimedia Foundation itself as a body. The foundation will intervene if there's more systemic harassment, something where someone would need a lawyer, something where if, like, more direct threats are brought in, that can be like someone threatens you up at your workplace, right? Like the foundation will step in there. But the foundation is extremely hands off with how harassment is handled, which my opinion is that's not necessarily a great stance to take. But it's this fascinating case that nobody knows about because nobody really talks about Wikipedia as a social network. And it's also important to highlight that there's over 200 different language wikis, and they all have their separate cultures and different rules.

And what that means is that there isn't necessarily an arbitration committee for every different. language wiki. So there's an arbitration committee for English language Wikipedia. English language Wikipedia is not Wikipedia. So Arabic Wikipedia, for example, is not an Arabic translation of English language Wikipedia. It's people writing an Arabic writing articles. Wow. And they're different articles. And does they have their own committee or is just one committee for all? So I don't know if they have their own committee because while I was at the Wikimedia Foundation, we were not given a grant to study globally online harassment. We were just given a grant to focus on English language Wikipedia. Wow.

So there's all these different kinds of rules. It's a very fascinating subculture. And a point I want to make, though, is that this has existed before and it doesn't necessarily work. And the size of the community on Wikimedia projects is much, much, much smaller. Also, then it's important to highlight what do you do on a platform? Because that's also part of the design of the platform. So do you consider editing in the commenting section of a newspaper, a social network?

It can be. You know, I would argue that the Kenja communities that came out on the Gawker Media platforms felt like communities, it definitely was when I was reading and writing a lot in the Jezebel comments. You'd see people that you noticed before. That's how it's a Gloria Ryan. I remember her as Morning Gloria. Yeah. She's now a very well-known writer, but she was a prolific commenter in Jezebel and then was able to pitch an article or two and then eventually join the staff. Right?

So, like, that is a social network. But what you do on that social network is different than maybe what you do on Facebook, right? Commenting on an article is a different kind of motivation space. And the design of commenting on articles looks very different than Twitter, for example. So I bring this up because what do you do on Wikipedia? What do you do on wiki data? What do you do in Wikimedia comments where you upload images? These are different kinds of designed actions than what you would do on Facebook.

And you're also more constrained in your actions than what you do on Wikipedia, because you're building an encyclopedia, and there are hyper-specific rules as to how you add content and how that content is considered neutral, how that content is verified. So the arbitration committee, while it has all of its faults, they're also solving an easier problem because they're just talking about how you edit something

and fights that maybe occur from editing and fights that are related to Wikipedia. That is so, so extremely different than what could exist inside of Facebook? What are all the different kinds of fights you could engage with someone on Facebook that could be brought up to the Supreme Court of Facebook, right?

Yeah. And just run them through your mind for a second. It could be posting misinformation. It could be starting a misinformation campaign. It could be impersonating someone. It could be sending dickpicks to someone. It could be soliciting child porn. It could be grooming minors.

It could be stalking your ex. It can be your violent father who's a drunk found you, right? There's all these different kinds of things and it would mimic maybe more real life in a way. Yeah, it's like, you know, think of all the ways that human communication can go wrong and they go wrong that way on Facebook. Tell me up next, here's some advertiser content from Dell Cinema about how binge watching has changed everything. Kayla loves TV. I like to tell people that I invented binge watching TV shows.

I'm in it for the long haul. And chances are you're a lot like Kayla. Over 70% of Americans are binge watchers, and they feel a deep connection to both the characters and the screens they're watching them on. Dr. Emel Steiner is an assistant professor who studies binge watching at Rowan University. With the newer screens that are now available

because of the crispness, the higher fidelity, they allow viewers to see a more realistic world. And that social realism creates greater feelings of connection with the people on the, those screens. According to Steiner, it's not just screen size and clarity that creates that deeper connection. The technology today allows viewers to control not just what they watch, but where and when they watch it. And this is great news for Kayla. I used to feel truly embarrassed about the amount of binge watching that I engage in, but I feel grateful that the culture is totally supportive

of this type of hobby that I have. If you're a person, who can never say no to one more episode, check out the Dell XPS 13 with Dell Cinema technology. For incredible sound, color, and streaming, it's the laptop for people who watch things on their laptop from Dell computers. Thanks to Dell for sponsoring the show. Learn more about Dell Cinema's amazing color and Intel Core I7 processor at Dell.com slash xPS 13. That's dell.com slash xps 13.

I wanted to shift gears a little bit and look at getting back to your piece that you did and this history that you've done. I wondered what your piece has to say about the way that harassment campaigns end. Do they peter out? Do platforms respond effectively to them? And is there anything that you would say about platforms that have responded effectively to to harassment campaigns over time and whether there are any lessons there? I don't think any platforms have responded well to harassment campaigns, because we would have seen that in a post-Gamergate world.

And one of the things that I'm hoping to do in a longer write-up, I'm actually, I met today with a friend and we're adding a lot of data viz to the project, which I'm quite excited about. And I still have to fold in all of 2017. And this relates to 2017 is how do you differentiate between a hoax and a harassment campaign? and does it matter? And at times I think it does. It does if you look at the intention. It also does if you look at how it's analyzed then. So, you know, there was a lot of sock-wetting happening and Operation Lollipop and your slip is showing. And in Father's Day was a hoax hashtag. And this was designed to harass. But what if you look at, for example, Gamergate, which started off as a harassment campaign and hoaxes appeared out of that. So later on, in the write-up, I talk about how a Sikh man he was photoshopped in to look like one of the Paris bombers. That was related to Gamergate. It happened a year later. And that was a hoax designed to harass.

So it's slightly different. It took on a much different kind of meaning. But it was a hoax campaign still under the umbrella of Gamergate. And so do these things matter? I think that they do when we start to get into misinformation and disinformation campaigns that end up producing harassment. There, for example, was one female reporter. She was Asian, and it was coming out of, like, something related to Trump or to a trial. And there was this woman in the background, like, taking a photo of something.

And Fortune was trying or Reddit was trying to identify her. And they kept misidentifying these different female Asian reporters. So this was a campaign that broadly you could, I think, define as a harassment campaign. It was, I think, related into this sort of umbrella of the alt-rate. But then it created micro-harassment campaigns against innocent people that were misidentified. So do we see this happening in misinformation campaigns? We totally do. I think this is where Pizza Gates is really important, right?

How many people were negatively affected by that? So many people's children, their images were used and misused, right? That wasn't a harassment campaign targeted to hurt someone in the way that in Father's Day was. but harassment was an outcome of this massive campaign. And so one of the things that I'm trying to think about how to argue, or rather I'm structuring the argument around now in the write-up, is that the difference of a hoax and harassment campaign and in the light of 2017.

And are these things like wildfires that just kind of burn themselves out over time when they use up enough oxygen? I would think the way to think about it is 2018 is the best metaphor to describe harassment campaigns. Something happened last week. something happened three weeks ago. When were the Kavanaugh hearings? Were they a month ago? Two months ago? How much stuff has happened since then, right? That's the same thing with harassment campaigns. The reason they peter out is because something else happened. Or like it maybe it did just sort of wind down. But then something else occurred and in people's lives, not just like another harassment campaign popped up, but like something else happened and the internet turned its attention to that.

So let me propose an intervention that I do think has been effective. So something that a lot of reporters noticed over the past year and a half, two years, was that in the immediate aftermath of any calamity shooting, Wadthyr, something like that, you would go to YouTube and you would search for it and you would immediately get a bunch of conspiracy videos explaining that there was some nefarious cause to the calamity that the mainstream media wouldn't tell you about. And because of YouTube's recommendation algorithms, people would often stumble onto these videos even if they weren't looking for them. And so an entire genre of content just became searching YouTube after calamities, and we would write about all of the garbage that was floating to the top. And, you know, after a while, YouTube got wise to this and now very closely monitors those search results in the aftermath of those calamities. So they've sort of spun up a team that can intervene and that can promote mainstream coverage that tells people what is actually happening. So, you know, I would point to that as an effective intervention that can catch some of this stuff in the moment and stuff that, you know, might otherwise start as misinformation

end up in harassment like a PizzaGate scenario, they can hopefully nip those things in the butt. But those aren't happening with harassment campaigns, though, right? Those are happening with things like wildfires, but we aren't seeing that happen with harassment campaigns. So let's talk about that. So let's say that someone was like an individual citizen, whether they were prominent or not, and they were targeted by some sort of harassment campaign. In your mind, is the right thing to do for the platform to go in and sort of strip out all content to stop surfacing hashtags or like send hashtag search results to an empty set of search results. Or have you started thinking about what in your mind is like a fair and effective way to target some of these harassment campaigns?

Sure. I think a big thing is probably them looking at their policy around how they're defining online harassment and considering rewriting it. Another thing I think is maybe like, again, as you suggested, lessening some of those or sending the search results to an empty space. but I think the biggest thing is probably having a team that can help them understand much more quickly, like, that this is a harassment campaign. I know from a lot of my work with different platforms that a big thing that they're worried about is online censorship and how online censorship relates to online harassment. And that's also a very American stance to take on looking at content and the ways in which companies decide when to take content down or not. It always falls back to. Is it censorship or not?

well, you know, we can't engage in censorship. But they do. They take things down all the time. They take things down all the time that shouldn't be up there for a variety of reasons. I think the bigger thing is having a deeper understanding internally as to what is considered harassment, or are the different levels of harassment that they are willing to allow on the platform and not, and how culpable they feel with that harassment. Well, but to bring us back to a point I was making earlier about Twitter, like Twitter's harassment policies are like relatively clear. They ban all kinds of harassment. The problem is they don't enforce it, right? or they enforce it extremely erratically.

So, you know, I'm open to the idea that rewriting the policy would be a big help. But to me, I see the issue as being much more related to their ability to enforce their own rules. For sure. And I hope that my answer wasn't saying that this is just a policy answer. It is also enforcement. It is having a team that can understand enforcement. It's also making sure that team, which is probably content moderators, has the contextual understanding as to what is happening, that they have good tools to take down content, that they, for real, though,

like some of the tools that people are given to content moderate from like what we have seen and like different like leaked information about content moderators is awful. What they have to work with is really, really not great. And it's upsetting if you think about it from a design standpoint when you think about how the beacons of modern design in the United States will be software design and it will be from major platforms that what content moderators have to have to use to moderate platforms is. really antagonistic. Outside of the content that they are forced to look at, the tools that they're given and the timeframes that they're given to analyze content is, I want to argue, like, a human rights violation. Wow. They have under 10 seconds to make a decision. Sometimes what they're looking at is extremely violent content. They have to look at it all day, and they have specific quotas that they often have to meet. So how do you build context out of that? How do you build context out of something like a harassment campaign that's happening in Steubenville? How do you build in context? to that. What people often have is like a checklist that they have to memorize. It's based off policy that's not great policy around defining harassment, and they have to make a split-second

decision. And they have tools where there is not enough shown to them to perhaps understand what they're looking at. So, you know, what are all the solutions here? The solutions are perhaps redesign the way in which content moderators are engaging with content on the platforms, make that experience better? It's hard to provide suggestions right now, because there are lots of platforms that use different tools, and they're all designed differently. You know, so that can be,

do you have the original text that was used with it? Do you have the text that was before the event occurred and after? Do you have, if it's a video, for example, do people have time to look at the video? Are they responding to something that's in the commenting section versus the video? Is there another video they have to see?

You know, is this video in response to another video? Yeah. All of these things, like, how would you lay out a product to do this? Well, you could have multiple videos being shown. You could try to, depending on how, like if you have reporting, right, someone's filed a report, ask them, ask the reporter if they want to highlight another video in a specific section. Can people have a couple minutes to look at something?

That one, to me, seems the most insane. Yeah. So, like, it's thinking about stuff like that. Like, the problem is not just policy. It's also, like, how do you enforce harassment? Well, who are the people that enforce harassment? Yeah. It's content moderators without them realizing it, too.

Like, they're looking at flagged information. So I think a really big problem, again, always comes back to design. So I want to mention something that I want to ask you one last question because you've already given us so much of your time. It's funny as you talk about moderation and pulling stuff down, we ran an excerpt of Sarah Jong's The Internet of Garbage on our site right before she went off to the Times. And the section we picked was her talking about the intersection between content moderators. and copyright law, and how the copyright law has stepped in to form the content moderation function because fundamentally there's money underneath it and there's like liability, and that creates, we're actually just going to pull this down.

And then you look at a platform like Amazon, which has many problems of its own, but it's an economic powerhouse and there's money there. So there's literally like Amazon litigators that exist. So if your product gets kicked off the store, you can like hire an Amazon lawyer and they will go fight Amazon on your behalf. none of that seems to exist on the regular, on the sort of like consumer platforms, right? There's no YouTube moderation lawyers. There's no Twitter moderation lawyers.

There's not this economic incentive like humming along in the background to make sure everyone's a good actor. Can you replace that incentive in some way? Well, there are lawyers that have a lot of expertise in social media, especially with now online harassment lawyers. So Carrie Goldberg immediately comes to mind. She's actually based in New York. She's one of the most well-known revenge porn lawyers, I'd say, but also just in general, the person that I think of when I think of online harassment policy and social networks and Danielle Citron, for example, right? She's a law professor, I think, going to Boston University right now from the University of Maryland. So there are lawyers out there with a specific expertise around online harassment, right?

Asking about economic incentives, I mean, we can look at how Twitter was not purchased. And I think one of the main reasons that people kept citing was the toxicity of the platform and also like rampant harassment. Yeah. So even if it doesn't feel like it, I think that there is a dollar sign on harassment. There are economic incentives. One of my running jokes is I tell the CEO of this company that we should buy a Tumblr for a dollar because like why not? Like what else they're going to do with it? And he's like figure out what to do with the porn and we'll talk.

It's like this like it's like a half joke that we tell each other constantly. But he's not not joking. I mean, but some of the porn on Tumblr. is good. Fair enough. You know, I mean, but yeah, I think harassment has a monetary bottom line, though. And as you've just pointed out, right? Like, why wouldn't someone want to buy Tumblr?

Well, a variety of reasons, you know. Why wouldn't someone buy Twitter? Well, I can think of a couple of reasons why. If Facebook were for sale, would you buy it? Right now, yes. Right now I would, just because I'd get to fire some people. But that's, like, that's me. But, like, you know, how do we, if we're thinking of these companies,

which I also want to argue are part of social infrastructure. So their worth is not actually something I think we should put a dollar sign on. They function as social infrastructure. So that's my next question, and this is right about it and now. The beginning of this piece, we focused on the timeline. The beginning of your piece actually sets up these definitions. Like, here's what harassment actually means. Let's come to a shared definition.

These companies are under no obligation to share their definitions with each other. Right. So it would be great if Facebook had an intern. consistent rubric of harassment, and they were teaching their moderator as well. But if you're just a regular person and you're using three platforms every day and YouTube's set of definitions are wildly different than Facebook, you're kind of hosed. Is this where the regulation begins? It can be. One thing I will say is I'm almost 100% positive that every platform has consistent internal definitions of harassment because they have, all of them have good policy team members.

So that's something that they define because that also does drive the different kinds of technology that they build in a way and the different kinds of design choices that they make. So they have clear policy. What's different is do users understand how that policy is enacted when interacting with content?

And that's where people don't understand, right? Because it's not necessarily clear in the outcomes as to what's taken down and what's kept up and why. And that's not just a content moderator's fault. A lot of that is also just that like the policy doesn't at times make sense. and that does differ platform to platform. So, you know, there could be some kind of legislation that says, like, platforms have to make public their internal policy choices.

I would hesitate to say that we should then have one overarching legislative decision around how to define harassment because what we would end up with is a very U.S. focused definition and view of harassment. And that would be implemented upon so many different countries. So, for example, Germany looks at hate speech in a very different way. Like what you're allowed to sort of post that falls under hate speech is different than the U.S. You cannot post Nazi insignia's online, for example. France, for example, has different rules around how children are protected. I think they were in the first countries to implement this right to be forgotten that Google then had to implement.

But that's not implemented on every single product or in every single country, right? So it's important to recognize that, like, the definitions we have in the U.S. U.S. around online harassment, should those be definitions that, for example, the EU has to implement? Or should we be following the EU, which is also another question to ask? Like, should we be looking at GDPR regulations and thinking about how that piece of legislation is going to have long-term design effects and technology effects as well as policy effects? And should the U.S. or should social networks, which are technically based in the U.S., should they be looking towards the EU, around policy decisions.

These are the kinds of much more high-level questions we have. Once you have the information of how does YouTube view harassment versus abuse, is abuse more extreme? Do they have another word for an extreme version of harassment or a less extreme version of harassment? And how does Facebook view that or Twitter? And then how do they view it in different countries? Is a country going to write a law that holds all of them to a certain standard? Do you think it should be our country?

No. No, I really don't. I always hesitate to say that, like, regulation is the answer. But I don't think that this is something that should fall in the U.S. I also think right now, under a specific administration, would not be a good time. This is what I always bring this up. Like, do you really want these people writing that law right now? No.

But I also don't think that a law would necessarily solve this. Because then you have deeper questions, which is what Facebook and Twitter and Google are facing right now with the GDPR of, well, how do we actually process what your law is requiring? I think there's something specifically in it around certain content has to be taken down or analyzed within 24 hours and then taken down. That's actually not at a pace that most companies can work at regardless of size. So like that stipulation, for example, does that benefit anyone? Yeah. And I think that's specifically data breaches, right, Casey? Sure.

Yeah. Let's all guess it what's in the GDPR right now. I feel like I should know. I'm like, don't hold it against me if I get this wrong, but I'm like 75% it's data breaches. Okay, last question. Okay. We talk about this a lot. The Verge cares about a lot.

I think our listeners care about a lot. I often feel paralyzed, and I'm in a position of, like, some authority. Like, I can see a harassment campaign directed against a verge person and be like, that's dumb, stop it, right? And, like, 50% of the time it stops. But I still feel paralyzed all the time. What can the individual actor do in this moment? I think it depends. I mean, there isn't one answer.

There's not one answer to rule them all. But let's go small for a step. Let's say you see harassment happening against. someone. There has been a lot of research done into what's called the bystander effect. So someone that has equal privilege to the one engaging in harassment saying to the harasser, hey, like, I don't think that's cool. Or why are you doing that? Or, hey, you should stop. There's been a lot of studies done on how actually this is useful, that this is a way to build better norms. So if you feel paralyzed by something and you want something to stop, you should try saying something.

Yeah. You should try using your privilege for good. If you want to take it a step further, I think now is a great time to organize and to look at like-minded groups or create a group that is maybe demanding for certain kinds of action that you want to see on platforms. I think protest and organized protest is shown to work really well. For example, in Berlin, Google is trying to open an office and a bunch of local anarchist groups and protesters got together and made enough noise and protested for many months. and Google is not opening this one big office there. They have other offices. But I think this is a good example of direct action working. I wonder how long Laura is going to stay at Twitter, you know.

But imagine if a bunch of people that didn't agree with Twitter's policies actually organized outside of Twitter's New York offices and San Francisco's offices and were just consistently loud or we're just consistently there and showed up and handed out flyers and said, I have a problem with hate speech on your platform. I think that that would deeply affect Twitter because you're in, you're close to their space. So I think that there's all different kinds of actions that people can engage in. Yeah.

Well, we'll see how long Laura lasts train to the door. I mean, it's really cold. It's also, like, it's also insanity. Katie Notopoulos from BuzzFeed tweeted, like, I hope Laura is in it for the long haul and is wearing a diaper. She, right before he came on air, she was definitely talking about it, being herself. So I don't know about the diaper. She's clearly in it for long haul. It is one of the craziest things to happen today.

I'm almost sorry to end it here, but in the context of this very good, very serious, very nuanced conversation. It's a perfect way to end. That's also just happening right down the road. Caroline, thank you so much for joining us. I really appreciate it. You've obviously done just an incredible lot of work on this. I have to come back soon. Thank you.

Yeah. Thanks for having me. All right. I want to thank Caroline Cinderers for joining us in the Burgecast this week. Quick Laura Lumer update. She got cold and just asked the police to cut her handcuffs so she could go home. She's still banned from Twitter, of course. If you are really interested in this topic, I want you to check out the episode of Why Did You Push That Button that button that we put out this season about why people delete their tweets?

It's a really interesting knock-on effect. Harassment campaigns exist. So in response, people have started using paid tools to delete all their tweets. Ashley and Caitlin did a great job with that episode, so you go check it out. We'll be back later this week with a regular Vergecast. And next week, super special surprise that I can't tell you about. But it's coming. And now another word from our sponsor, Dell.

The Dell XPS13 with an 8th-gen Intel Core I-7 processor is the laptop for people who never say no to one more episode. With lifelight color, brilliant sound clarity, and smooth streaming, Dell Cinema technology makes whatever you love to watch even better. Call 800 by Dell to learn more or visit Dell.com slash XPS13. Sponsored by Dell and Intel.