Technology, Connected - The Hidden Human Cost Of ChatGPT - Empire Of AI

Episode Date: September 16, 2025

AI data annotation, RLHF, and the hidden labor behind ChatGPT are the focus of this Thinking on Paper Book Club episode on Karen Hao’s Empire of AI. We examine how OpenAI’s race to scale relied on scraped data, content moderation, reinforcement learning from human feedback, and low-paid workers reviewing violent, sexual, and abusive material to make AI systems safer for everyday use. The conversation connects Sam Altman, OpenAI, GPT-3, Microsoft, model safety, stochastic parrots, AI exploitation, disaster capitalism, Venezuela, Colombia, and the human cost of building commercial AI.--Other ways to connect with us:⁠Listen to every podcast⁠Follow us on ⁠Instagram⁠Follow us on ⁠X⁠Follow Mark on ⁠LinkedIn⁠Follow Jeremy on ⁠LinkedIn⁠Read our ⁠Substack⁠Email: [e-mail]--🕰️ TIMESTAMPS(00:00) Trailer(02:00) Introduction to Empire of AI & Karen Hao(03:41)Shifting power dynamics in Silicon Valley(03:59) Karen Hao’s warnings in Empire of AI(04:56) Humanity V the relentless race for scale(06:32) The environmental impact of AI systems(07:38) Stochastic parrots: Silencing Critics(09:48) Sam Altman Loves A Military Quote(10:53) What Cost Humanity?(15:14) The global race for AI advancement(18:32) The hidden labor behind ChatGPT(25:07) The ethical dilemma at the heart of AI development

We are reading Empire of AI by Karen. Can you scale indefinitely and have the good of humanity at heart at the same time? No, of course you can't. If it was for the good of humanity, they would have guardrails in place before. Somebody kills themselves because they use that technology. I want you to rate Mark's spiciness today from zero is bland as hell. Ten is fire shooting out of your face. Power dynamics.

It's about employing hundreds of thousands of people for pennies to sift. through some of the most grotesque, ugly, violent sexual content that's ever been created by humanity to feed the language models. Growing as much data, no matter how shitty, how toxic, wherever it's data, we're chucking in in this model because more data, more chips, gets us where we want it, gets us where we want. And the Struggle came out with a reference that one model's training impact is the equivalent of nearly 5,000 flights from New York to San Francisco. The demand for data was going so vast that countries were scuba.

scraping whatever they could find on the internet, inadvertently capturing Mautox in abusive language as well of subtler, racist and sexist references. What did they do, Jeremy? They stopped openly publishing their service. Storm up the Trowbridge! Yeah, lock it down, boys, like it down. It's classic, classic Godfather.

I thought the workers did to help each other at the platform pitied them in competition. Projects were first-con, first-serve, and a task stuck around in queue only as long as it took for enough workers to claim it. This window between a task's arrival, and its disappearance shrank over time from days to hours to seconds as more and more workers, including many Venezuelans in crisis, joined and vied for scraps of work. Chapter 9. Disruptors and Curious Minds, Booklovers. Welcome to The Thinking on Paper Book Club. I am Mark. This is Jeremy, and we are reading Empire of AI by Karen Howe. We're on part two,

and it's about exploitation. It's about power dynamics. It's about employing hundreds of thousands of people for pennies to sift through some of the most grotesque, ugly, violent, sexual content that's ever been created by humanity, to feed the language models, to make them safe, to make them child-friendly, to make them what you use every day to day. Jeremy, how do you feel reading some of the chapters in here, Venezuela, Kenya, Colombia, exploitation? Chapter 9, disaster capitalism. I was very troubled, very disturbed. It's almost like when you watch a gnarly horror movie and your brain scrambled and you feel very weird and uncomfortable

and unsettled and ungrounded. That's the way I walked away from that chapter. This has been happening for years, though. This is not AI specific. In chapters 6, 7, 8, we get more of the same, the rise to power of Sam Altman and Open AI. Scale, scale, scale. We get the commercial AI.

applications, we get chat GPT3, the paranoia of open AI, the lockdown mentality. They were fearful of their model weight's been stolen. There's a lot of paranoia on the rise up. So we can go through that first, for sure. Yeah, but the mechanics of the rise, the rule of the game of Verizon are to have the most compute, do it the fastest, push product out the quickest, no matter the cost. That's the race. Competitive advantage, competitive advantage.

Stay ahead. stay ahead, stay ahead, stay ahead at all cost. Damn the torpedoes, right? Damn it all to hell. Damn it all. Mark is feeling very spicy today, but I think there's a reason for the spice. What do you want to roll with? Let's start with Ascension.

Let's go through the philosophy to 10x everything. In the race to the top, powered by a fear of losing, the race makes someone more susceptible to compromising values. Can you do the right thing in scale wars? Is it possible? And, you know, the flip side of that, you have. Safety being viewed as product prevention, anything that prevents 10x in this case is bad. The rules of this game that they're playing are not for humanity.

Safety and ethics being a product blocker is a sad state of affairs. One of their goals stated in the research paper was to understand neural networks and deep learning so we could figure out how to use the tools we have or figure out what the tools we have can do. So you're building and building and building and building, oh shit, what can these things do? Well, your question, you said, can you scale indefinitely and have the good of humanity at heart at the same time? No, of course you can't. Sam Altman, he's got this quote by Hyman Rickover, an admiral in the US Navy. It's always an admiral in the US Navy with these people.

They think they're doing something protecting, I don't know, the people. And the quote, the future of the human race, blah, blah, blah. And Sam Altman wrote below this, building AGI that benefits humanity is perhaps the most important project in the world. We must put the mission ahead of any individual preferences. Low-stakes things should be low-drama, so we can save our high-drama capacity for high-stakes things. It's like he's trying to convince himself with redundant quotes that what he is doing

is for the good of humanity. If it was for the good of humanity, they would have guardrails in place before. They wouldn't react after somebody kills themselves because they use that technology. Oh, we've got that wrong. We'd have to sort that out. after the event, after the event.

Oh, but Mark, wait. You know, so when... Makes me so angry. When GPT, I think it was GPT2 first came out, they only released the 8% version. So that was their safety protocol. They only released the 8% version. Hey, friends and neighbors in the chat,

I want you to rate Mark's spiciness today from zero is bland as hell. 10 is fire shooting out of your face. Last week, Jeremy, you said I was wishy-washy, and I thought, right, no more wishy-wish-wish. Washy. Way to take a stance, Mark. Way to take a stance. So a couple other things noted in here. Just kind of confirming what I call the rules of the game in this race, make money to do more research, not do more research to make more money. What did you think of the simple little minus to plus

mistake? Someone pushing the update actually had a minus sign flip to a plus sign in whatever code he put in. And basically all the output was lewd and offensive. And imagine just like a flip of a certain switch or missing of it, you know, talk about not having your eyes. eyes dotted in your T's crossed, that turned into a bit of a shit show, didn't it? Did that person get a pay reward or did they lose their job? That's a fair question. It could be, it could be looked at both, especially if you're looking at it coming from the product prevention department or the grow at all cost department. You also had the, again, another serendipitous or accident was the scraping of Reddit,

which gave them all the coding capability that they hadn't really been planning on. There was actually some interesting emails from people in Open Aigs. Oh, maybe we should pay some of these people for all of this information that we're stealing from Reddit and GitHub. And maybe Microsoft. Is it plagiarism? I don't know. Let's keep it rolling, right? Well, there's the thing about maybe Microsoft can pay them after the event.

And Microsoft gave like, I don't know, 10 pence to charity or something. Yeah, check the box. Well, here, listen, so let's put our corporate hats on it for a minute here. You know, where it's easy to be the independent guys on the outside going, man, you guys are, you guys are, you guys are doing the world right. But you also, if you're, if you're scaling a company and you have the resources, you, you tied your, tied your boat to Microsoft a bunch of money, a bunch of compute. Hey, if the rules of the road, if the rules of this game we're playing are scale at all cost, you, you got Microsoft, you got access to the biggest amount of chips, compute, the whole nine yards. but then with that Yeah, that's all

Yeah, just don't tell us that humanity is at the center Yeah, totally. With my corporate hat, I agree with you completely. Because Microsoft's like, I gave you all this stuff. What are you giving us?

And hey, tapping my foot, yo, when's this return coming? But again, like you said, outward facing, we're saying, hey, we're designing something

that's going to benefit all of humanity. Yeah, are you really? You can't do both. But of course, if Microsoft are dangling another two billion in front of them,

which they were for the API to be a success. So of course, yeah, from a corporate perspective onward. Yeah, a couple of things in six, and then we'll move on to seven. This is where they start talking about the API for GPT3.

Let's get this thing out the door. Let's get as many people using it. The more people that use it, it creates this flywheel. So let's push that product out there. P.S. The initial live version with this API didn't have any content moderation filtering. So, hey, we're still in the race. We're abiding by the rules of the race, not the rules for benefit of humanity, but the rules

of scale. You mentioned code generation. I would almost argue that code is more valuable than text because code teaches something how to move through a logical process to make a decision. That's this big aha moment. And Microsoft's like, hey, Sam, we've got GitHub. There's like billions of lines of code. Why don't you grab this stuff?

Oh, wait, wait. But isn't that a free online community or an online community that people are doing it for the benefit of humanity and sharing their code to help other people code and create models. Oh, damn it, you know, just grab it and we can get really smart, really fast. Scale at all cost, right? Let's see. What do you think of page 155, how to destabilize an organization? That was Peter Willinder.

He was originally in robotics for OpenAI. Eventually he became the VP of Product, and he was talking about the internal discussions that were going on. He says it was like the US Intelligence Manual on non-violent sabotage, where there was just using, you know, talk as much as you can, ask questions that don't make any sense, refuse to understand any of the logic, refuse to understand any of the question. We spoke about last week, the petulance, the immature nature of this, the school ground, one-upmanship. On that note, we're talking about an empire, military. A lot of these founders, a lot of these very famous CEOs and businessman that we label as saints, they quote from

military leaders as if it's a carte blanche to do what they want. So there is a tie in, I think, between big business and the military. One aspect of the military is there's a mission at the top and there are people in between to cascade down and influence and rally everybody in lockstep towards that mission. So what does a company do? Very similar. The mechanics are pretty similar, which is why they're pulling from that stuff. Let's get serious, Jeremy. Yeah, so we go into chapter seven science and captivity. This is where we start talking about carbon footprints of training models and Emma Struble came out with a reference that one model's training impact is the equivalent of nearly 5,000 flights from New York to San Francisco. The first time someone started

putting their hands up and going, what's this stuff doing to the environment? All right. Page 165. Ooh, what you got on 165? If anyone in the chat has read on the dangers of stochastic parrots, can language models be too big? This was the paper that set the ball rolling in, in, in, in, in, terms of what could be the harmful effects of all of this. In short, it outlines four key warnings. As you mentioned, the energy impact number one. Second, the demand for data was growing so vast that companies were scraping whatever they could find on the internet, inadvertently capturing more toxic in abusive language as well of subtler,

racist and sexist references. Third, because such vast data sets were difficult to audit and scrutinize, it was extremely challenging to verify what was actually in them, making it harder to it. eradicate toxicity. And finally, the model outputs were getting so good that people could easily mistake its statistically calculated outputs as language with real meaning and intent. This would make people prone not only to believing the text to be factual information, but also to consider the model a competent advisor, a trustworthy, confident, and perhaps even something sentient. And that's been a theme we've been talking about for months and maybe years. Like Google ended up

pulling the paper and not allowing her to publish it, silencing the things that need to be said. But she stood her ground, didn't she? And she fought and she fought and eventually. Yes, not tied in with Google though. On that, so there's just a quote here. Open now I had simply admitted in its research paper describing the model that GPT3 did indeed entrenched stereotypes

related to gender, race and religion. But the measures for mitigating them would have to be the subject of future research. And again and again we see this after the event, after the fuck up, after the toxicity, after everything that we're trying to eradicate, they go, oh, we'll sort that out later, or God, we should have sorted that out before. Well, here's the Phil site. Where was the foresight? Where was it? Mark, rules of the game. There's no time for that. We're buying GPUs. We're throwing...

But humanity, Jeremy, humanity. We're throwing as much data no matter how shitty, how toxic, whatever, it's data. We're chucking in in this model because more data, more chips, gets us where we want to... gets us where we want to go. And you know what? Here's Terry, a military collateral damage, Jeremy,

this is maybe their thought path. And if they read enough military, not propaganda, but whatever, then eventually collateral damage. Yes, some people have to suffer because we're saving humanity.

Yeah. God complex. So the tail end of this chapter, there's some reference of like, I thought this was pretty interesting. Karen Howard references, you know, big AI going big tobacco.

Yeah. So, hey, hey, these cigarettes are, are great for you, dude. It's all good. We're not going to publish anything about that they'll kill you in the next, you know, eight to ten years. But now it's all good because it looks really cool. Check out our ads. We're great. Is it this like the Aaron Brokovich, the beginnings of the Aaron Brockovich movement against big tobacco then, yeah, against big AI.

Yep. So conveniently after, and this is where it bumps around because we're going from GPT3 to now they're referencing chat GPT. Right around chat GPT timeframe, guess what OpenAI did according to the book, according to Karen's research. What did they do, Jeremy? They stopped openly publishing research. So, remember a long time ago? Draw up the Trowbridge. Yeah, lock it down, boys, lock it down. Listen, listen, here's, remember earlier on, it was like, hey, if someone else gets where we all want to go before we do, hey, we'll all hug up, we'll all make this thing happen, let's all do it together.

But no, like you said, the drawbridge is closing. But guess what? It's classic, classic godfather. The Corleone is right. Lock it down. I don't want the family speaking to anybody. I don't want anybody outside the family coming into this room. I want no discussions beyond the Corleonis.

That is it. Lock it down. How's your Godfather character map looking these days? It's got a lot of blood on it. There's a lot of victims. All right. So while they stopped publishing, openly publishing research, Stanford has a transparency tracker,

which I think that's pretty cool. I didn't know about that. So that's a good little tool to see who's doing what. The dawn of commerce, Mark. This is post-Aidé divorce, new research direction. Guess what? We want to have an aligned AI system with vastly more capabilities or vastly more capable. How do you do that?

I would say there's three steps to that. Okay. I would say there's three steps. First, scaling GPT3 by another 10x using a new supercomputer from Microsoft arriving in the third quarter with 18,000 Nvidia A1 the newest, most powerful GPUs then exist. I think you need that first. Then maybe doing more research to increase by 25X open AI's compute efficiency. And third, I would say, would be improving the quantity and quality of training data, in part,

by tapping into user data and shifting the model towards the best parts of the data to distribution with reinforcement learning from human feedback. And let's deploy as many models as possible as products. Flywheel, flywheel, flywheel. So, yeah. They were really getting the hang of it by this point, weren't they? They were like, yeah, we've got this. We've got this.

There was a bit at the beginning, 2016, 2017. They weren't sure, you know, Microsoft may be going to invest a billion. But by this point, they're flying. So aligned AI system, I mean, we talk a lot about AI alignment, you know, with human and humanity kind of stuff. It's hard to do when you're still operating in the scale, in the race to scale. Like they're diametrically opposed.

I do agree from a technical standpoint that, you know, after investing billions in compute, maybe you can look at, you know, how to get them a little bit more efficient. Oh, for sure. Use what you got a little bit better. But the quality of training, this is when we start bringing in the RLHF,

reinforced learning, human feedback stuff that will continue to talk about, quote, one of the things and things in the research initiative, the new, hey, we're going to organize now that Dario Amadeh is left, we're going to reorganize. One of the goals, Mark, of my new research organization, we're going to study the science of deep learning a little bit better so we can understand how what we're building works. We're going to, let's do that. Does that sound like a good initiative?

Can we catch up to the train? Sounds like something I might have done a few years before that particular moment in time, but hey, I'm not, I'm not them. So I did like the quote, I use your thing, quote, scaling multimodal models, meanwhile, could potentially quicken the pace of improvements even further with enough breakthroughs, the document said with remarkable definiteness, we might actually reach AGI. Models feed into models, it quickens the process. So from the business perspective, again, makes sense?

I have a question, I have a question. What's AGI? That depends, doesn't it, Jeremy, on whose definition of AGI? It's like moving the goalposts. It's this one week. It's this the next week. It seems to be becoming more attainable the more the definition is weakened. Yeah, I asked that a little sarcastically because the definition, like you said, is all over the place.

But all right, so let's dive into the muck of this last chapter. Two words that seem very non-threatening. Oh, data annotation. We're doing data annotation. Hey, I'm going to hire a bunch of people. We're going to do some data annotation. Oh, sounds kind of official. But what this actually is is looking at content, whether it's text, video, pictures,

whatever it is, and labeling that content. So there was a case study of guy in here. I definitely want to reference his name because I think is important. Moffat O'Kimyi. Okay, so what he had to do is review 15,000 pieces of explicit content per month. I'm going to let that sink in for a bit. Like, what does that do to your mind? But that was the most disturbing thing to me was what reviewing that kind of thing, even if it's text-based, does to your mind over time.

And for him, it pulled him away from his family. It pulled him away from his wife. It pulled him away from a lot of things because I don't think many people could do that and come out on the other side unscathed. And by the way, they were getting paid pennies on the dollar to do some of this really dark work that it wasn't just explicit content. It was the most explicit content that a human being could create the most nefarious evil human beings. It's talking about parents raping their children. It's about children being raped by animals. It's about the most grotesque things the human imagination caught with.

And on it every day, thousands upon thousands. And they mentioned, like, his marriage breaks down and his wife leaves him because he becomes so inside himself that he just can't function in a relationship properly. I wrote a note underneath disaster capitalism. And I put, I'll read it takes my handwriting's shit, but the opposite of clockwork orange. It's a conditioning, the opposite of clockwork orange. So remember the character that had, you know, had his eyes wide open, he had to watch a certain content to change his conditioning. Like, this is changing mental conditioning. This is changing your identity, your personhood, how you think when you got this, like, just shit fired at you consistently.

Like over months and months and months and months. And again, nobody, I would argue not many people can come out of that. Maybe nobody can come out of that unscathed. And what makes it worse is that the exploitation of it, they had no choice. Many of them thought they had no choice. They wanted to keep doing it at the beginning because they wanted the money. They couldn't live without it. You're literally holding them to ransom.

You create this feeling of need. It's almost like they're saying they should be grateful. Karen Howe runs down a whole kind of theory, I guess, that finding these crashing economies, these countries with crashing economies and going, man, we need to go here because these guys will do just about anything. And unfortunately, you get situations like Venezuela where the economy is like crumbled. The inflation was like, I mean, 10,000, whatever the heck it was. It was like it wasn't working.

They couldn't get food. They couldn't provide for people. on one side, these companies are like, well, hey, we're doing them a solid. But are you like, if you're doing them a solid, like, let's pay them for what they're doing. Or, I don't know, man, it's praying. It's praying on folks. And that was. Exploitation.

Again, it's for you've got to scale and quote for all the workers did to help each other, the platform pitted them in competition. Projects will first come, first serve. A task stuck around in queue only as long as it took for enough workers to claim it. this window between a task's arrival and its disappearance shrank over time from days to hours to seconds as more and more workers including many venezuelans in crisis joined and vied for scraps of work and then you have fueness saying and despite how much stress and hair-pulling it was causing, she couldn't imagine leaving the platform. Quote, Fuentes taught me two truths I would see reflected again and again among other workers who would similarly come to this work amid economic devastation.

The first was that even if she wanted to abandon the platform, there was little chance she could. Her story as a refugee, as a child of intergenerational instability, as someone suffering chronic illness was tragically ordinary among these workers. Poverty doesn't just manifest as a lack of money or material wealth. It seeps into every dimension of a worker's life and accrues debt across it. It's, it's, it's horrifying. When some of these workers started using automated tools to help them respond to things quicker,

they were identified and booted from the platform. What they're doing is building a version of that for people more fortunate to use. You're going to tell them they can't use tools like that, but they are helping build a tool that. it does essentially that. Define irony. I know this is pretty dark, but it's important to highlight this piece of the puzzle. Like when you're using chat GPT or you're using some of these tools,

there are people behind the use of these tools that are getting paid, nothing to do like mind. Horrendous work. Mind altering stuff in a bad way. Well, that gets you thinking twice on using some of these tools? I mean, I don't think anyone's going to not use some of easy. button because of that, but like, I don't know, can there be an accountability? Can there be a, no. If the last week of thinking on paper hasn't caused pause for concern, I mean, if we're not,

if we don't do something, we can't really expect most of the population to do something. Like, it's, it's pretty, eye opening is not the right word. I know it transcends that. It's transcendental. I don't know if you've ever read any of Robert Celdini's books. Yeah. Presuasion. Is that him?

Pre-suasion. And there's a lot of psychology behind helping someone. I think there was a study in that book where there was an apartment complex in a rough side of town where a woman was outside literally getting stabbed and murdered and screaming for her life and everyone's sitting in their apartments thinking someone else. Oh, someone else has got to be helping. Like I don't need to go out there. Someone else has got to be helping.

And it's one of those things where we have to scale the help. So on the flip side, let's balance. Let's balance the scales, right? So the scale on one side is just AGI or bust, damn the torpedoes. And then on the other side, it's how do we scale what we're talking about? And the way you do that is by people coming together and talking and then more people coming together and talking and more people coming together and talking. It's a lonely road. But like, hopefully as you walk through these towns, maybe another person walks with you and then another person walks with you.

And hopefully we're creating some awareness in that because, man, it's not a popular message from a business. No, it's not a popular message. AI has a very happy, smiley face, but behind that face, there is a lot more that meets the eye. Do you think that this is the end of reinforcement learning from human feedback? AI will increasingly take on that role itself. Have we seen the end of the darkest days, or is the worst yet to come? Those are you not listening live? If you have a question and you're reading this book, comment in the chat as you rewatch this

or send us a note, hello at thinking on paper. xyz and we'll respond and have a chat we're learning as well this is basically open source this is real time learning jeremy real time putting our reputations on the line putting our thought patterns on the line i like it same yeah it feels good uh hope you guys are enjoying it as well we have all of our stuff up at thinking on paper dot xyz instagram lincoln youtube there you have it well food for thought guys hope everyone has a great weekend uh don't have nightmare Thinking on paper.xy Z. I'm Jeremy. This is Mark. Be curious. Stay disruptive. Keep thinking on paper.