Your Undivided Attention - A Facebook Whistleblower — with Sophie Zhang

Episode Date: July 9, 2021

In September of 2020, on her last day at Facebook, data scientist Sophie Zhang posted a 7,900-word memo to the company's internal site. In it, she described the anguish and guilt she had experienced over the last two and a half years. She'd spent much of that time almost single-handedly trying to rein in fake activity on the platform by nefarious world leaders in small countries. Sometimes she received help and attention from higher-ups; sometimes she got silence and inaction. “I joined Facebook from the start intending to change it from the inside,” she said, but “I was still very naive at the time.” We don’t have a lot of information about how things operate inside the major tech platforms, and most former employees aren’t free to speak about their experience. It’s easy to fill that void with inferences about what might be motivating a company — greed, apathy, disorganization or ignorance, for example — but the truth is usually far messier and more nuanced. Sophie turned down a $64,000 severance package to avoid signing a non-disparagement agreement. In this episode of Your Undivided Attention, she explains to Tristan Harris and Aza Raskin how she ended up here, and offers ideas about what could be done at these companies to prevent similar kinds of harm in the future.

Structurally, we don't have a lot of information about how things operate inside the major tech platforms. Most people who lead these companies signed non-disclosure and non-disparagement agreements, so they can't speak publicly about their experiences. But Sophie Zhang was different. As a data scientist at Facebook, Sophie stumbled across a pattern that was going underneath the radar. Leaders of small countries, like Honduras, Azerbaijan, and Albania, were exploiting a loophole

to make them appear more popular than they actually were. In 2020, on her last day at the company, Sophie blew the whistle. She posted a 7,900-word memo to Facebook's internal site about how foreign leaders were exploiting the platform. She then turned down a $64,000 severance package in exchange for the privilege of speaking out. Today, on Your Individed Attention, Sophie shares with us how she came to feel responsible for the political futures of countries that she'd never even

visited, and what she thinks can be done to prevent that kind of exploitation going forward. I'm Tristan Harris, and I'm Azaraskin. And this is your undivided attention. So I joined Facebook from the start, intending to change it from the inside. I mean, I was a friend with them about that. I told them, I don't think Facebook is making the road about your updates. That's really told my recruiter. That's precisely why I want to join.

Like, if a company is perfect, there's nothing to fix. I'm joining it precisely because it wasn't perfect so that there are sense to improve, that they can make a difference. And frankly, I hadn't anticipated how much of a difference I would actually be able to make. What was their response to that when you said, I don't believe Facebook's making a better place and I would like to make a difference? I think they would like, you'd be surprised how many people at Facebook say that. Because ultimately, Facebook is, where this historically was a very open company,

like open dissent about the company was allowed and encouraged we had weekly question and answers with the CEO. I don't want to make it sound like the entire company was dissidents. If you think Facebook is amazing, you are more likely to want to work for Facebook. But overall, Facebook was a rather non-hierarchical company. It presents its surface very open. There was significant amounts of truth to that because dissent was allowed and to some extent accepted. I certainly wasn't shy about criticizing the company from inside while I was there. And as well, employees were able to easily speak with everyone else of the company, which I understand it's much more difficult at many other companies. With that, I know Facebook has made changes since I left. My understanding

is that they've restricted discussion of non-workplace-related matters. Obviously, I don't work at Facebook anymore. I can't say from first-hand experience. But I'm talking about my experience during my time there rather than since I've left. And it was also not very hierarchical. I was an IC4 at Facebook, which means nothing to the vast majority of people who are listening to this. So I'll translate that. I was literally one level about a new hire straight out of college. That's called an IC4, you said. Yes, I see extends for individual contributor as opposed to a manager or something. I was not anyone important within the company. They were like probably a thousand people with data scientists with my rank and they will only want more because Facebook doesn't have a ton

of data scientists. It's predominantly engineers, product, people, etc. But even though I was very low level, I was able to interact with leadership, worked on extremely important matters. Like, I personally briefed Guy Rosen, who's the company vice president of integrity on what I found in Honduras. And I want to highlight how unusual that was. It would be like an Army sergeant briefing Kamala Harris on something. It would never happen. And if it did, it would be emblematic of something very unusual going on. And so that is when strains of Facebook, So I'm sure they're second guessing it very much right now, that there was this culture of openness and non-hierarchical matters that I was able to jump in and work on these problems and try to fix them and have quite extraordinary access relative to my actual rank and title. We had lots of slogans.

Every company has slogans, and at Facebook, one of them is that nothing at Facebook is someone else's problem. And the idea is that if you see something wrong, you should try to fix it. you shouldn't assume that someone else is fixing it and is taking care of it. And they could always point to that and say, see, I'm just trying to live by the motto and the logo. And just to clarify from the start, because they want to be clear about what I did work on, what I don't to avoid confusion, that they worked on inauthentic activity. And by inauthentic activity, I mean fake accounts that are conducting various activity, which is separate from misinformation and separate from newsfeed decision making.

So I was on the faking engagement team, and so by fake, I mean, fake accounts, hacked accounts, self-compromised accounts in which people willingly give over their credentials to some nefarious actor, and by engagement, I mean likes, comments, shares, etc. There are teams at Facebook who work on civic integrity. There are teams at Facebook who work on what's called coordinated in authentic behavior, which is what the investigations they do for things like Russia 2016. My team was not in either of those areas. It was largely as spam team and by spam, I mean just the more general definition of things that are low quality, repetitive, not individually harmful, but harmful in large quantities in aggregate.

But you can also see how that was my ticket and excuse to working on matters that I considered important in my spare time. I was able to work on increasingly more sophisticated political activity and work closely with the teams that did work on coordinated in authentic behavior. So I'm going to use an analogy here. And that's the difference between an FBI agent and a policeman for a small village in, say, Idaho, sorry, Idaho. The average person might think they're vaguely similar, they both cops. But these are jobs that are very different in scope in organization, in importance and training, and if you compare the two of them out loud, the FBI agent might get a bit offended.

And so my situation was essentially that there was some local village policewoman not very important, except I kept finding things that went up to the FBI in this analogy, which would be very unusual, or have political consequences with internal politics, because on one hand, the FBI is happy that they're getting all this work done for free. They don't even have to assign anyone to do these investigations. On the other hand, they feel a bit embarrassed or upset. They're being shown up, and they're maybe unhappy that she's giving them extra work. Who should come over and give me more work? I mean, she's not my boss. She's not even in my organization. And meanwhile, her own village police chief is maybe a bit annoyed that she's doing all this FBI work instead of working on

village crime, which is where she belongs. And so that's an analogy I would use to try and describe the situation. Because fake engagement sounds very serious, but most of it is not that severe, and it's focused on high-volume scripted activity. And I was essentially petty crime department. That was my team, that was my role and my job. But my job was defined widely enough that I could argue that it was made sense for me to look into more importance, there were volume cases in my spare time. I never got training. To the extent the oversight and guidance I thought was stop doing this and work on your actual job. Everyone knew what I was doing. I talked about this up to a company vice president. It wasn't a secret, but it just wasn't officialized or

regularized. Like, I was essentially running the shadow integrity program. The way I'm understanding it is that the bulk of the problem, when you just think about fake engagement, is this sort of low-quality, unsophisticated spam. There's like lots of these little petty crimes. Then there's the FBI that's really focused on working on high sophistication, high impact fake engagement, things that affect world elections. But in between here, there is unsophisticated, there's high sophisticated, but then there is semi-sophisticated that sort of sits between the two and no one was looking at that. And that's what created the space for you as a sort of like town cop to move up and be like, hey, an entire swath of activity is being missed. I think that was part of that. And I think

the FBI agent would be annoyed at doing that in an energy, just like the teams would be annoyed at doing this in an energy because you're like, that's not fake engagement. Fake engagement is low level, just like if you compare an armed bank robbery to shoplifting, they're like, this isn't shoplifting, this is much worse. And so it's definitely the case that teams that were focusing on this were responding to cases that were more important. But I would also highlight a different aspect, which is that they're also responding based on PR considerations. And what I mean is that they're primarily reactive rather than proactive, at least in my time at Facebook. They weren't usually going out and looking for things if they did was mostly in contrast like

the United States. Rather, a lot of their time was spent looking at outside reports from news agencies, from opposition groups, from NGOs, saying this is weird going on, let's take a look at it. I would say the area that I really pioneered is that I was going out and looking for it on my own. There wasn't anyone who pointed me to this. I don't think if you asked anyone, pick two countries in the world that you think are most important. I don't think anyone would respond to Honduras and adzu pejorang, quite frankly. They're an awkward combination. But I think in the case, as you were looking in Honduras, they weren't even particularly trying to hide. I was literally just put up the person's Facebook page. I didn't know who Juano and

Ando Nandez was until I googled him. Oh, wait, he's the president of Honduras. Great, I guess. And then I opened up his Facebook page. I wanted to take a screenshot for a public point. I just wanted to open up the people who liked him and take a screenshot. And then I stopped because they realized something very surprising to myself. And that was that most of the people who were liking his posts, they were not people at all. They were pages pretending to be people. And so pages are a Facebook feature that are meant for businesses,

for instance, for celebrities, influencers, politicians, etc. People who have a public profile, they're a public-facing thing that's controlled by individual users. But this was a loophole at Facebook. There are smart people who are at Facebook go on and look for fake accounts. there was no one at Facebook, and to my knowledge, still is no one who goes out and look for pages pretending to be people. The analogy I use is that, for instance, jaywalking is illegal in much of the United States,

but it doesn't mean that the cops go out and find people jaywalking and stop them. And so this is a loophole that was used by adversaries throughout the world. It was used by the governments of Honduras and Azerbaijan. So some listeners might be thinking of themselves, well, how consequential is it? So if I can run 200 accounts and impersonate some public perception and sort of steer conversation, you know, on a few posts, this sounds like a pretty negligible thing. But yet you and your memo said that you felt like the stakes of this work were so high that you were losing sleep. You used the phrase that you felt like you had blood on your hands,

not in the sense that you caused something, but in the sense that whether or not you would intervene would be consequential for an entire country or for an entire opposition group or for an entire minority group. how would you talk about how significant this set of issues really is for those who are skeptical that this has any influence at all? So I want to be clear right now that it's always hard to figure out the uncertain consequences of these sorts of indirect effects. You can't say that this newspaper article caused X, Y, Z to happen by changing X opinions. I'm not an expert on politics.

I'm not an expert on public discourse. I'm not an expert on public relations. But if anyone is an expert on politics, public relations, and discourse in the nations. It's the presidents of those nations. Because you don't become the leader of a nation by being stupid and throwing money down a whole. If you stay a national president for more than a decade, you have to be very good at maintaining the support of the people and the support of the important people. And so I'm not the experts, but it's very clear that the people who are the experts have decided that this is worth the money and expenditure.

Like I caught the governments of Honduras, I call the governments of Azerbaijan red-handed. They weren't even trying to hide. And to me, that's very telling him, because I mean, Adiras, Azerbaijan, these are not the wealthiest countries. The presidents do not have a ton of money to spare. These people genuinely believe that this was worth the expenditure, the effort, paying the people to do this full-time, and the risk of being caught. And to me, that's telling, that's suggestive, the people who are the experts have spoken. The analogy I would use with the Russian interference on social media in 2016, which I think

was very shocking and jarring to a lot of people. And it's hard to say how much of an impact that actually had. I don't think that anyone can say for certain. And they think that there are always arguments that it wasn't that decisive, but I think it was still very shocking to the psyche of the nation. And what I found in Azerbaijan, what I found in Anduras was, I think, rather worse. Because in Russia 2016, that was a foreign nation. Like you expected that from Russia, for Honduras, for Azerbaijan, it was the own country,

it was the own president, their own government that were doing this to them. And to me, that made it an order of magnitude. force. When people imagine the sorts of consequences for this sort of behavior, I think they usually jump to the conclusion that this is intended to persuade or et cetera. But I would rather focus on a different aspect, and that's the perception of popularity, the perception of public opinion. When it became to how much it was able to do in my spare time, I think I took a step back and decided what my priorities were and what my standards, what my moral compass was. Like, for instance, I decided from the start that I would never act unilaterally. I would try very hard to avoid

being judged, jury, and executioner. I would only act in the role of investigator. I would always let other people confirm. I would always let other people sign off to take action. I would always let other people do the actual enforcement as long as they could help it. Because, I mean, honestly, I could have gotten away with asking forgiveness instead of permission in a number of these cases, probably. And Facebook will say, and they're technically correct in doing so, that I was never the only one acting. Like I said, they were always. always other people signing off. But in practice, I never received answers of no. It was always, yes, let's do this, or silence, no answer. And whether I got a response,

as far as I could tell, was entirely dependent on how much I yelled and made noise about it, and etc., that got a decision to be made, which was invariably, yes, let's take it down. In some cases, it happened overnight. In others, it took more than a year. In some countries, it hasn't even happened at all. Most people try to distance themselves from the consequences of the work. They tried to keep it at arm's lens. They do what they need to sleep at the end of the night. But in my case, I thought that there wasn't anyone who would make those decisions, who would take those responsibilities if I didn't, and so that's why I decided to take that on to myself, I suppose. Ultimately, it's hard to say how much

of a difference I made. It's hard to say how much a difference any of this made. That's the crux of the issue. I'm trying very hard to be cautious and be realistic about what we know and what we don't know. And for indirect impacts, it's always impossible to say what the actual consequence is. One of the other topics that comes up in our discussions around this area is what a friend of ours is called the gradient of privilege. That, as you mentioned, the number of days that it took to action content

on these other countries that are sort of further down the food chain in terms of priority. You mentioned in your memo that... So just a tiny clarification. I would clarify that this is actually in users and behavior, not content. Yeah, excuse me. So users in behavior. And so in the Guardian article that highlighted here, this is how many days did it take Facebook to address inauthentic behavior after an employee flagged the case? In Poland, it was one day. In the Philippines, it was seven days. In Taiwan, it was 11 days. India, 17 days. It was only seven days after I found the U.S. connection. There's a separate part for the Ovalon network. But anyways. Yeah, it depended quite a bit based on circumstance. It depended quite a bit on who the person responding was. was because there's a lot of variants. Like, what happened in Poland, wasn't that Poland was number one based on Facebook's list.

It was very silly. The employee looking into it was a Polish person. He was like, oh my gosh, this is happening in my country. I mean, I had flagged this a day after Christmas. I woke up the next morning to see that he had already taken it down. Actually, the policy people were a bit upset. We're bringing up such an important and fascinating point here because part of this issue is about representation.

If Facebook had employees that were, and there's still a question of like, why should they be the ones playing God for their own country, and does one person who's Polish best represent what should happen in terms of what accounts you take down or not in Poland? But certainly the fact that he or she cared about that is interesting. It's certainly the case that I think most people care more about their own nation. That's true for every country and for region, like the United States cares more about Britain and Western Europe because of cultural affinities, because of language affinities for the United Kingdom. And so Facebook's employee base is mostly comprised of a mix of American slash Western European people, plus Indian and Chinese employees here on HB and visas.

And so Facebook does face employee pressure on these points because, I mean, people care naturally about their own countries. And I think I'm very unusual in that I came forward. I quit based on countries that I have no connection to. And so to a large extent, Facebook has faced a lot of pressure from its employee base on countries like India and the United States, in some case more than the PR pressure, because Facebook employees are very difficult to implement, especially if they all quit on mass. And that's essentially a dynamic that privilege with certain countries over others. But even when countries are prioritized, that priority and treatment is often not what people within those countries would prefer, I think. Because like India is a very large country,

and it is a priority country in Facebook's metric. It's as important as the United States in Facebook's metric zero on the top tier. And so in India, when I found inauthentic activity that was being run for political activity, Facebook was much quicker to take it down. With the exception of a single case where we had gotten permission to take it down, sign off to take it down, we were about to act, and suddenly we realized that the network was run by an important Indian politician. This was a member of the Indian parliament, the Loc Sabra, who was honestly so arrogant was silly or stupid, that he hadn't even bothered using a VPN to run his fake accounts. And as soon as that was ruined, suddenly everything stopped.

I never got an answer or even acknowledgement. I never got a yes or no. I kept asking. And you can argue that, okay, you sent someone a message or don't respond. Maybe they just didn't see it, because, I mean, no, sometimes people don't respond to emails, they miss things. But when you keep asking and they keep not responding, when you're already in conversations with them about other matters, and you talk about those matters perfectly fine,

and then you add in, well, since you're here, what about this? And they ignore you, and this keeps happening. It's very clear that something is unusual is going on at that point. Ultimately, I guess Facebook decided that keeping this person happy was more important than protecting the sanctity of the world's largest democracy, and they disagreed with some strenuously in that regard, and I think that most Indians would also have a different opinion, because, I mean, importance, again, and decision-making is taken through the lens of Facebook's self-interest,

because, of course, Facebook is a company. The U.S. probably gets, in the U.S. elections, probably gets the most amount of Facebook's resources, attention, bot scanning, you know, like... They absolutely do, but a separate point I want to raise is that Facebook has more the way to act independently in countries that are less important. Very interesting. And what I mean is that, like, Facebook was defying the governments of Honduras and Azerbaijan, may say so. If I had caught the government of India doing this, they would have absolutely never would have done the same. they would have had long discussions and then found out some reason to say why they shouldn't act or something like that. The government of India is quite powerful.

They can threaten to arrest Facebook employees. They have threatened to arrest Facebook employees if Facebook doesn't go along with their desires. Like if the government of Honduras bans Facebook, then that occurs on theuras far more than Facebook. Quite frankly, his own citizens would be very unhappy. But India does have a lot of leverage and power over companies like Facebook. They can demand that Facebook essentially sent employees to be paid. based in their own nation to be served as potential hostages, essentially. That's actually what I understand they're proposing right now,

that India has actually asked to make, not as they're passing on, I think, laws, to make sure that social media companies have physical on-the-ground representation from employees, and then to, after that, to impose criminal liability if the companies are not acting them. So you're actually talking about a very real scenario, essentially taking hostages so that you can get your way. Yes, absolutely.

Because on one hand, countries that were less important, they got less priority and attention. But the lack of attention also meant that decisions could be made with less political interference that things got made even when the government disagreed. Sometimes the lack of attention can be a benefit. It's double-edged sword. In a sense, there really is a playing God around the world,

and sometimes that God is beholden to a certain power or interest, and then sometimes that God can just be, Sophie Zhang, you know, making a decision on behalf of some of these other countries. And I can imagine just the feeling of weight and responsibility that is reflected in your letter. Certainly. The usual defense that Facebook has is that it's hard to find since. But I think part of why precisely my story and my work was so powerful and spoke to so many

employees was that I had done all the hard work for them already. I had already found the bad people. And yet they sat on it. They aren't really disputing anything that I have to say. They can't because I know I'm speaking the truth. And because I've documented it already extensively. In Honduras, what we found from the same thing, started, it was very obvious what was going on, and especially after I realized that whenever

the people controlling hundreds of these fake pages, pretending to be real people, was one of the page administrators for the Andurum president. This was someone who had been specially trusted and charged with making statements on the Honduran president's behalf to have access to say things for him, and that's what it did in. And so it was very clear from the start that this was a troll farm that was run by the government of Honduras, or someone authorized by them. He took this to the Facebook leadership in my presentation, like, I said, and the response I got was very surprising to myself, and that was a policy leader calling from some very Latin America, he told us that he wasn't surprised, because he had already

been told socially by people who ran social media for the Andyrian government, that they had been tasked with running troll farms on the Anderan government's behalf. And at the time, I was very shocked that they would just go out and admit something like this. In retrospect, I think that should have been my first right flag that he had been told this. I mean, they'd known about it already. I was very naive from the start. I thought, oh, Okay, I found this. I'd hand it over. They'd take care of it. Everything will be great. I'll go back to my actual job. Instead, it was the start of a fee and two-year ordeal. It took almost a year to take down the Andeuroan government's operation. They came back almost right afterwards. In Azerbaijan, the Zari government was also running their own domestic troll farm. It took more than a year for that to be taken down. As far as I know, they're both still going on. In terms of potential solutions that I would offer, what the companies can realistically do,

and might be vaguely in their self-interest to. I've suggested two separate things. The first of them is this. So at Facebook, and at other social media companies, I presume, but I can't speak about them just about Facebook from my own expertise. At Facebook, the people who are charged with adjudicating cases of saying, do we act in this case? Do we take down this person? Do we take down that person? There's the same people who are charged with maintaining good relationships with governments and important figures. which creates extremely obvious conflicts of interest. And Facebook will argue this is, I mean, it's a company, this is in its own self-interest, is important to, etc.

But, I mean, it's also the case that most news organizations are for-profit publishing and organizations, and most news organizations do have separation between the editors who decides what pieces get published and the people who are charged with having good relationships with governments. If a piece would make a government angry, then they'll have. have the PR people handle it, but they don't have the PR people say re-retain it because it will make the government angry. And so the news media has managed to at least have that theoretical separation. And I think it would be beneficial for Facebook and other social media companies to have that as well, even if I don't see it happening anytime soon. The second is more broadly

just that I think many companies like Facebook that rely a lot on metrics can be a bit short-sighted in their scope. And what I mean is that Facebook cares a lot about. its numbers, about improving its profit or etc. But the issue with numbers is that if you only pay attention to metrics, if you emphasize them, you naturally de-emphasize things that can't be measured. And I think Facebook has suffered in this area to some extent because it's been very bad at proactively managing its PR. I don't think it's controversial to say that existing Facebook public relations, this public sentiment on Facebook is absolutely terrible. And so Facebook does have incentives to get ahead of negative PR to prevent problems before they become fires.

But, I mean, preventative measures are much harder to justify than reactive, because it's much easier for a fire department to say, we put out 10 fires last year. They can't go and measure. We had fire safety codes that prevented 100 fires from happening last year, because, I mean, how do you even measure that? You can't. And so the first seems tangible. The second seems wishy-washy. And so I think Facebook has difficulties with that to a large extent, because frankly, Facebook is a company that acts in its own self-interest. So you left in 2020, yes? I was fired in September of 2020. So you've been through a lot. You've seen a lot. You've blown the whistle. I'm curious for the other sophies that are still within Facebook or within the

twitters or the TikToks, what advice would you give them? I don't want to tell people what to do. Like, different people have different considerations. I didn't have to feed a family. I don't have children to support or starving relatives that they needed to support back home or something. And so I don't want to begrudge others the decisions they have to make to keep the lights on for themselves. Everyone has to sleep at the end of the night and how we achieve that is up to themselves. But for the people who come in idealistic and do want to make a difference, I would tell them to think very carefully about what your goals are, what your motives are, and where you would draw the line, because I think working within the system to change things is

difficult. It's very powerful because you have an extraordinary amount of access, but at the same time, people naturally like the people they work with better, they get used to the situations they work with. And it's easy for people who come in from the outside who want it to fix the system to get used to the way things work until the supporters feel betrayed. And so I would ask people to be very clear about what they want to achieve, what their goals are, how things would be received upon and seen by the outside world, and what your own priorities are, what your goals are. I would tell them that as a new employee, I managed to catch two national presidents

right-handed and made international news on multiple occasions. That's what I did, what can you do? Do you want to stand in solidarity with tech workers like Sophie? Join the Center for Humane Technology and a coalition of other aligned organizations for a special event to build solidarity among tech workers and tech users on Tuesday, July 27th. For more details, visit HumaneTech.com. Your undivided attention is produced by the Center for Humane Technology. Our executive producer is Stephanie Lenn.

Our senior producer is Natalie Jones, and our associate producer is Nur al-Samurai. Dan Kedmi is our editor-at-large, original music and sound design by Ryan and Hayes Holiday, and a special thanks to the whole Center for Humane Technology team for making this podcast possible. A very special thanks goes to our generous lead supporters, including the Omidyar Network, Craig Newmark Philanthropies, the Evolve Foundation, and the Patrick J. McGovern Foundation, among many others.