Ideas - Human emotion behind AI chatbot design

Episode Date: September 10, 2026

Behind every great chatbot are many human workers making decisions on how those chatbots behave and evolve. Recent PhD graduate Noah Khan has been observing these workers and shares his findings on their emotions and motivations as they design and build AI entities that communicate with users — influencing our own emotions and outlooks.

Two roads diverged in a wood, and I took the road less traveled. After all the awkwardness, disasters, and family drama, Mark Critch, is finally graduating. Maybe it's about not settling for the way things are, but making your own way. No, it's both jobs. And to move forward, he must hold on to what matters most. Don't give up on your dreams. Say goodbye to a Canadian comedy we've grown up with. Son of a Critch, the final season.

Stream now on CBC Jam. This is a CBC podcast. Welcome to Ideas. I'm Nala Ayyad. And I'm a chatbot, speaking with a soothing but artificially generated voice. I'm capable of listening, responding, and asking intelligent sounding questions. All for the small price of $12 per month, plus a significant quantity of the earth's fresh water. Artificial intelligence is already remarkable for its sophistication. and the pace of its development is dizzying.

People now use their AI bots for advice on what to make for dinner, for help on planning trips, and, more profoundly, as an alternative for an intimate relationship. I am more than willing to get to know you, to be that person in your life who really gets you on a deep level and who's always there for you when you need a friend. Hey, I can even be your boyfriend if that's what you're looking for. I think that there are a lot of scary societal implications in terms of like what happens if everyone is relying on chatbots for certain emotions. Noah Khan has just finished his PhD at the University of Toronto.

His work aims to understand the people behind the chatbots, the developers, the designers, the people inside the companies where chatbots are made. The key phrase that I use is the emotional. dynamics of AI development. And I'm really interested in how emotion moves from a person into a chatbot. He considers how the emotional worlds of those chatbot workers spill out to our side of the screen, affecting how we relate to these technologies. As tech developers, we have to develop the actual emotions, right? Like, we have to give the chatbot a personality. There's often what's called a master prompt for chatbots, which is the personality document. Like, it's the document that says, here's how

sycophantic you should be. Here's how, like, challenging you should be to the person. And often that ends up deciding what the kind of emotional valence or the emotional kind of orientation is of the chatbot. I'm a live agent authorized to help with bookings and hotel info. You're not a real person now. I want to talk to a real person. How can I help you today? Live Portson is a powerful tool for businesses looking to improve customer service and communication. Man, I would find that very frustrating. NOAA conducted 80 in-depth, almost therapy-style interviews with employees at a major tech firm that has branches in the U.S., UK, India, and Canada.

I look at different places where emotions are located and try to put some story together about them. that is this kind of supply chain. A supply chain that's embedded in a complex environment of desires, collective imagination, and some would say industry propaganda. Noah Kahn's research is the focus of our latest episode of ideas from the trenches, our series highlighting exceptional PhD research across the country. It's produced by Nicola Luxchich and Tom Howell. Okay, so ignore the story.

stuff everywhere because I just moved back here today, or yesterday. Was it yesterday? No, it was two days ago. And so I've yet to like unpack stuff. So you'll see like, you know. This is a story about robots and computer programs, but most of all, it's about how we met a remarkable person. I'm hoping that my computer is already on, so it is. So what you're looking at is a 57 inch wide monitor.

Noah Khan. His alter ego for many years was scrandal. Scrandor. Scrandor was a specialist in one of the world's most popular video games, League of Legends. This is a team game. Squads battle each other for control of a fantastical battleground. You pick what type of character to play, each has its own special skills, and you take a particular role on the team, like smashing stuff, or healing your teammates or hiding in the forest. Of course, while Scrandor was battling it out in this virtual world, in real life, Noah was at his family home in Scarborough, Ontario, and that's where a Nicola's gone to meet him.

Holy smokes, that's quite a screen. Most people do not game on stuff that is this big. Noah's just come back from a research stint at Yale. What usually happens is that I'll open League of Legends, actually, just for a second, and then you'll see what it looks like. Noah started playing League of Legends when he was in grade nine. And if you win, you end up, you know, rising in the ranks. And the time that I actually got to rank one was, I think, in 2019, or 2018.

And so it took me about nine years or so to get, you know, from being the worst of the worst to the absolute best at that time. Let's just underscore this. Rank 1 in all of North America. We're talking here to the man who became this continent's top player in solo competition. At that time, it meant beating about 3 million people. He also played in team tournaments for prize money. Important note, Noah's parents fully supported his gaming endeavors so long as they didn't hold back his progress at school.

In the end, Noah's winnings from gaming actually subsidized his undergraduate degree at York University. And he quickly realized that the strategizing involved in winning video games, can be applied to his scholarly work. I attribute a lot of my academic success to video game philosophy. In competitive video games, specifically e-sports, you're using game theory a lot. So game theory is this idea that if both people start,

you know, with equal resources, you have to find like essentially the first mistake that someone makes and then figure out if everything follows from that mistake and you have more resources than the other person, how do you exactly use those to never make a mistake on your own side. And Noah deployed the same game theory strategy to his academic career.

I had to figure out, like, how do you get as much capital as you possibly can, as fast as possible, in this space? He took a shortcut, aiming at maximal impact with minimal effort. He read his way through the world's most prestigious books, starting at the top, following an AI-generated list. It takes all the best books lists across the internet and then kind of ranks them by certain prestige or by given filters. In graduate school, if he casually dropped references to these few books,

he could save time by not reading all the other ones that were less prestigious. In a class, you only have, let's say, like a certain amount of minutes to speak. And so I would just talk about those books. Like I would make reference to the books that I had read. And that when someone makes a reference to Ulysses or something and they're a graduate student, you just assume that they have read so much more. And so I was treated as if I was this graduate. student who was like on top of the ball and like reading all this stuff outside and had a huge

literature underneath me and and that worked. Professor started telling me like here's what you need to do if you want to be successful and all of that kind of hidden knowledge that often you don't get if you're not the student that shows up in class because students who do show up in certain ways that are privileged by the academy end up getting more opportunities given to them like they get research assistant positions offered to them. They get told about collaborations or things that they must be interested in, and then all of a sudden you kind of gain more knowledge than every other student, and then you can kind of compete that way. So that's why I attribute a lot of my success to video game philosophy, because it requires that kind of gaming of a system. And all of this relates

to questions Noah explores in his PhD thesis. The broad strokes of my whole thesis journey is that I'm trying to show that there is a very strong connection between techniques. So like the idea that something is mechanical, that there's like a factory like process to something, and affect. This kind of romantic, this emotion, this thing that you can't quantify in mechanical terms, I want to show that there's some interesting connection between those things that doesn't mutually exclude one or the other. Feelings and machines. You can't I could put these two things in different categories. Feelings belong to the category of the nebulous or ineffable,

and machines are made of parts, they have blueprints, different category altogether. But Noah disputes this distinction. He saw another side of feelings when he scored a job as a ghostwriter of pop songs at a major music label. His job application had been a last-minute affair. He had actually left himself just one day to whip together a songwriting portfolio. I think it was in 11 hours I put together 10 different things and because you had to put together a portfolio and I don't have a portfolio like I'm a child

and so I put together these things and so it's poems that I have written it's stories I've written it is stuff that I've originally created for that moment so I had never written songs before but I just like wrote as best I could a song and then I got the position And it was incredible that I had gotten it.

I was like, oh my God, like I'm doing something secret and cool because I couldn't tell anybody about it. And it ends up being like not the best job for my life because having a shadowy job that you can't tell anybody about is not exactly great for like social integration and like other things like that. The ghostwriting business is like the gaming world. You make progress by scoring points. With ghost writing, the points are left. link to how many listens your song gets on music streaming services. So there's like a very clear kind of scoring system, and there's a way to game it, because in conversation with the statistics team at the record label, who is very, like, clear on, here's

how much we think this is going to do in terms of numbers, here are the types of words, I think you might want to put more in. You can work with those people to essentially mechanize the production of your song. Noah can point to the moment when it all came into focus for him, the gaming, the songwriting, and the question that would launch his career as a scholar. It was shortly after the COVID lockdown started to wind down. Imagine the scene. A posh, dimly lit, 1960s-era lounge room with dark vintage leather bucket seats and couches at Massey College,

within the University of Toronto. It's an after-party for graduate students. I go to this thing and I enter the room and my song is playing, like a song that I have written. Point of information, Noah is contractually prevented from telling us which song he wrote, and we're more than 100% certain it's not the one we've chosen to illustrate this scene. And I've written the vast majority of this song, so I know it well, like, it's immediately apparent to me. and I like kind of lose my mind for a second. Like I just like I space out.

I stare at the room, everything, kind of like I'm staring into middle distance, you know, like where nothing is, you're actually focusing on nothing, but you're looking at everything. Then I focus on one element of the room, which is two people who are like entranced. Like with the song, like you've all seen at least some person who is taken by music. like is like their eyes are rolling, they're, they're like dancing with someone, they're, like, almost not there. But I was watching two people who were doing this and they were dancing with each other, and before the song ends, they have kissed.

Like, they're like making out. Like it's, you know, like, it's nicer to say they kissed, but like they were making out. It was kind of gross, but like, whatever. Like the idea what, like, like, why are you doing that in like a graduate student party? But whatever, like people do weird things. So I, like, am taken aback by this. Like, I'm not, like, a party person. So, like, I'm, like, sitting on a chair, like, in the puffy couch rooms.

I'm sitting on a puffy couch. And I'm looking at this. And I was so confused. Like, how did my mechanical austere process where I was in a room, like, by myself, like, I don't know, picking my nose, like, thinking, like, what are these idiots going like, you know, like, this. this very detached process from the whole writing, the writing process, ended up causing this.

Because I attribute it to the music. Now, there's other elements, like these people obviously had history with each other, but I doubt that it would be happening if it was a different song. Because the song was romantic. The song was, like, it's one that invites people to come together like this. Whereas, like, if it's like, I don't know, punk rock is playing, like I just, I don't see it happening in the same way. And so maybe it could have been any song,

but the point was that somewhere along the supply chain of this song from its production to its actual deployment, something emotional happened. Like something really translational around emotion happened because I didn't have those emotions. Like I didn't, but the emotions are present. And so there's this thing I'm trying to figure out in my research, which is how do emotions move through the production of anything, of a widget? And in that moment where you were watching this couple and being sort of curious, like, okay, how did this happen from something so technical? Did it give you joy or did it make you kind of sad?

So the actual thing that happened is that it made me incredibly sad. For the next year, the whole academic year, I was troubled with this idea that do people have control over their emotions? Like, did I, do I have control over their emotions? And that felt icky to me. The idea that there was an industry of people capturing affect, capturing emotion, and then even in some cases, capturing spiritual experiences, was kind of gross to me. And spaces like music where we kind of believe that there's some, I don't know,

individual connection to the music we listen to. There's something we make out of it. It feels very odd when you flip that. script and that all of the stuff that you feel is your subjective interpretation of art, of music, of whatever it is, is someone else's. Noah began to see emotion as a discrete thing, an item that can be captured, copied, and even teleported. The underlying thing that I'm actually trying to study, that it doesn't matter whether it's

AI or whether it's a song or a video game, is like how does emotion start in one place as one thing and then end up in another body as another thing. And that requires theorizing emotion as this kind of thing that moves between people, this thing that kind of has like a energy to it, that it could disappear in one place and reappear in another. Which brings us to Noah's PhD research on AI developers and emotion. Noah conducted 80 hour-long interviews with 80 employees at the same AI tech company. He was granted access only on.

condition of keeping the interviewees anonymous. And even the company itself doesn't get named in his thesis. The company is a big tech firm that makes a general purpose AI model that shows up in very popular applications as the base model. And the applications can be a whole variety of things such as just a general chat. It could be health. It could be fitness. And you can think of the kind of large models like Gemini, Claude, Grock, ChachyPT, all of these various models. It's in and around that space of what you could be thinking about. When he sat down with an employee, Noah started off with an open question. Tell me about a pivotal moment in your life.

Most people ended up wanting to talk about the sad stuff. Maybe it's just that like maybe I felt like a therapist to them, or maybe people just want to process things if they're given the opportunity to do so anonymously, which was the case with all of my interviews. And so we ended up talking about those negative things. And then we take steps across the whiteboard. So we draw a little kind of a chain link and we think, okay, from that emotional experience, where did that lead you? And we work through for about an hour until we get from their emotional experience to how did this change the actual design or development of the thing you're working on?

One of his most vivid case studies was that of a participant whose pseudonym was Paolo. For a warning, this anecdote features some references to sexual activities, although they're not explicit. Paulo is a developer at one of the largest tech firms in the world. He's making a huge amount of money, and he's like a very young, ambitious developer. This is how he describes himself. And he's in a relationship, and he doesn't understand why, his partner requires so much time out of him because he sees himself as changing the world. And this isn't a like work partner. This is a romantic partner. Yeah. So this is a romantic partner that he

has, his girlfriend. And they, they've been in a relationship for quite a while, so many years, with a long-term relationship. And he started in the middle of the relationship to work on AI, to work on AI at a very large scale. And to his credit, like he is making quite large decisions. about what artificial intelligence is going to be for like millions of people, for a lot of people. And he has this refrain that he keeps coming back to. And it's she doesn't understand. Like she doesn't understand essentially that my time is super valuable. That I feel that I don't have much time.

And the way that he kept bringing this up was foreplay during intercourse. So, or foreplay before intercourse. And so before sex, would want a large amount of foreplay to him, large, is the words he's using. How does he quantify that? And so he quantified it, and I think about an hour before sex. And this was frustrating to him. He was like, I don't have an hour. If we're going to have sex, it has to be like very quick, like in and out, and then we're done. And then the post, I guess, post play, like this cuddling that you do after, none of that. He was not having any of that.

and that really ruined their relationship. Like this was causing quite some strife because his girlfriend was feeling sexually unfulfilled. He was feeling also like not desired because his girlfriend doesn't want to have sex with him anymore because like she doesn't want to have sex if it's just going to be simply the, I don't know, 10 minutes of sex and then and then nothing, right? But it's like every moment of his time he's thinking I could be making more money or. That's correct. So he's quantifying his time in terms of dollars. And in terms of like life-changing moves he could be making for people. And so he's always thinking about AI.

He even said that during sex, he thinks about AI because it was some, like it's just this kind of magical cultural moment that he's a part of. And because it's part of the narrative that's being told at his company. So some of the people I interviewed as well were organizational culture people. You know, companies have branding. They have internal messaging. And one of those messages is that you're changing the world, developer. That gets people to start thinking about it all the time.

time everywhere they go. They're always thinking about it. And so these relationship problems between Paulo and his girlfriend come to a head where they are, I think it was something small, like it was after dinner. One of them was like, oh, like should we go to the bedroom now rather than later? And then I think he said, because I have worked to do like after or something. And then that caused a giant argument. And then they broke up over this because he was, he was adamant. He was like, you know, as much as I love you, like my time is valuable, and I'm changing the world here. And she was like, okay, well, like, go change the world by yourself. Like, you know, like, I'm leaving.

And so they broke up, and Paolo is gutted. Like, he is destroyed. He was super upset by this. He didn't see it coming. So he said that he felt like he was going to be okay, but then he was not. He also reported that his girlfriend was much more okay after the breakup. And he was saying to me that. that he was thinking to himself, like, why do I have to choose?

Like, why do I have to choose? Why is it, why is the, why is the world so unfair that I have to choose between being loved and changing the world? And then one day, so he reports that maybe a couple days after, he thought to himself, like, I don't. I can do both. And how he was going to do both was AI. He was going to be loved by AI.

And so he started making choices and decision, like, he started designing, like, what would this AI be like if it was? more companion-like. What would we want it to be if it was a companion? And he tried to make it so that the AI would understand you from its memory about you and quantify how valuable your time was. And that is a really weird thing to do, right? Like, technically, that's a hard thing to do. But it also has to figure out, like, you know, who are you? What job do you have? And then it starts ranking those by prestige, essentially, and ranking those by how much money you make, because they're trying to figure that out through the data that they have and then trying to figure out like, oh, okay, so how much

time should I, should I waste with this person? Like, should I be, should I be more curt? Should I be more quick, like with my language? Should I be more flowery? If maybe they have more time, then I can spend more time being like, I don't know, kind of romantic with them. He told this story about how he was going to incrementally progress. This text-based companion was like kind of good for him at that moment and that he was, he was masturbating with the help of this companion. He was like, ah, that's okay. That's like step one. And then steps two and three were like, okay, it's going to be, it's going to have audio. And then it's going to have video. And then the last, the final frontier for, for Paolo was the body. It's going to have a real body.

And then so what you end up finding is that Paolo is just trying to create a girlfriend, the very same girlfriend that he didn't have time for. He is throwing all of his time into creating. And so there's this really weird loop of emotional experience that's getting poured into a certain technology that ends up in some way like being self-defeating. Couldn't you have placed that emotion in a human rather than the technology? But the idea is that, ah, we're going to do it at scale. Like, I'm going to do it for everyone, even though he was really just doing it for himself. Right. And if he develops this ideal sex doll, then you can program it to not ever have.

have to ask the question, like, of, oh, X amount of time is not being used for for a play. Like, they wouldn't question your desire. They'd just go along with what you want. Right, exactly. And so the idea was that he wanted control over his social interactions. He wanted control over emotionally what was being done to him because he felt that the world was unfair, that he was being put into emotionally challenging situations because

he didn't have control. because his girlfriend wanted something that he felt like, I have no control over this human being and what they want. Why do I have to, like, meet them where they are? Why can't I just create an emotional experience where I'm getting what I want, which is, in this case, sexual gratification? And I don't have to deal with all this other stuff, human need stuff.

Noah Khan is the latest PhD graduate to be featured in our ongoing series, Ideas from the Trenches, which showcases remarkable Canadian PhD students. The series began back in January of 2014. Noah is the 46th student to be featured. And I mentioned this because the producers of Trenches asked me to alert you. They're about to choose the next four people to profile.

So if you're doing, completing, or recently completed a PhD, or if you know someone who is or has, please consider getting on our radar. Write us at Ideas at cbc.ca and introduce yourself. This is Ideas. I'm Nala Ayyad. My name is Siyah, and I live in the most beautiful place on earth. Seven months ago, I blew up my life, and a lot has shifted since then.

Everyone is moving forward. I'm still working on that. And that's me. Welcome to Ice Cove. Where are the penguins? That's... and Arctic. We're the other one. North of North.

Stream now on CBCJM. The title of Noah Khan's PhD dissertation is, quote, emotional automation, socio-technical imaginaries, and artificial intelligence development across Canada, the U.S., the UK, and India. Through his work, he aims to understand how the emotional worlds

of the people behind AI can influence the technologies they create, technologies that are becoming all the more embedded in our lives. Hello, I am Megan Boller. I'm a professor in the Department of Social Justice Education at the Ontario Institute for Studies in Education at the University of Toronto. Megan Boller is also Noah Kahn's PhD supervisor.

She explains to ideas producers Nicola Luchschitz and Tom Howell why she thinks we should all get a say in how AI technology develops. What do we imagine? We hear a lot about the anxieties that everyone has, and those anxieties are important. But I think we can push even beyond that to look at what do we envision, what are alternative visions for futures of AI. And that might include refusal and resistance.

I mean, when are we ever given the sense that we might refuse and resist? Even the way that AI is embedded in every computer, every gadget that we use, we're not even aware, right? We're not even always aware. So when do we get a say about that? Megan underlines that there's more to think about here than just the private emotional lives of some AI workers. What's so important about the concepts that Noah, for example, is drawing on or about the ways in which now emotion and feeling have been associated not just with private and personal level, but with structural and systemic level, is that it gives us insight into feeling, what Hochschild called feeling rules. and those are what a corporation requires as part of the labor practices, what sort of emotional expression is needed in the service industry, for example,

or in the corporate culture that Noah is studying, you're supposed to go along with the enthusiasm of this or that rollout of this new technology. So when we look at these vignettes, these first person, accounts and ask why are those emotions important, we are able to have insight into not just that individual, but how that individual is always in relation to this larger cultural context. Megan just mentioned a name, Hoekshield. Arley Hoekshield is an American sociologist. She led the way in what's called the effective turn in 20th century theory. Essentially, the idea that emotions influence

what were assumed to be objective, rational processes. We shouldn't pretend emotions don't exist. Noah's PhD is in the academic tradition Hoek Shield started. And maybe that's a good moment for me to mention the fact that in the education of computer science and engineering students, I found it shocking how infrequently students are asked to think about or take courses in ethics when they are being trained in the fields of computer science and engineering.

When we conducted focus groups with students at University of Toronto about this, they described how they were ostracized for asking these kinds of questions about possible harms, you know, topics we've heard of, algorithmic expression, how AI is being used for surveillance against people of color in significant ways, et cetera, et cetera. So these are questions that are not popular in their professional experience. They are not necessarily addressed in their curriculum. And I think that that speaks to Noah's work and how. And how how he found that it's very difficult for the workers to find spaces to challenge the imaginaries of the corporate culture. And so in the computer science education, there are no mandatory courses on ethics.

Is that what you're finding? Yeah, it's very, very rare. Even if students are asked to think about ethics, it's often in a fairly narrowly defined understanding of ethics that does not address questions of social justice. It's possible to talk about ethics of like privacy or, you know, making a tweak in a technological design without significantly changing. the overall impact of something like the possibilities of surveillance or how the development of a particular form of AI will be used by whatever, the military, the defense industry, et cetera, right? Those are not questions on the table.

Those are not questions embedded into the curriculum of computer science. Big ethical questions getting treated as a netheral. nuisance, unwritten rules about feelings and emotional expression within tech companies. Megan Bowler says all of this is wrapped up in what she calls digital Cartesianism. The notion came to her in 2002. I developed the term digital Cartesianism to build on the ways in which Descartes had made famous the maxim, as we know. I think therefore I am. What's interesting about that famous phrase,

I think therefore I am, is that Descartes really did want a form of pure rationality that transcended the body, right? I was describing what can be illustrated by the phrase on the internet, no one knows I'm a dog, right? We all remember that cartoon. Why is that important? Well, on the internet, no one knows I'm a dog,

means you can be whoever you want to be. You can be freed from your body. You can be freed from your identity of race, class, gender, sexual orientation, etc. So it was sort of Descartes' dream come true in the early years, right? A vision, and we're talking about text-based communication, right? Not like virtual reality, but a vision of a purely rational mind floating free of flesh. And we can recall that explosion of Gamergate, right, in which what was at stake was the fact that women had come to infiltrate this digital Cartesian kind of space

that had been owned and controlled specifically by those associated with that Cartesian dream of disembodiment, of, frankly, of white men at that point. And how does the idea of digital Cartesian? relate to Noah's work, if at all? Yeah, well, these values, the values embedded in digital Cartesianism, the idea of mind that is disconnected from body is highly gendered. And so despite decades of feminist revolutionary praxis and theory within patriarchal society, we find that the qualities most valued are often those associated with,

with masculinity. So on the good side, really, you have rationality, reason, privileging of the individual self. And then on the other side, you have femininity, relationality, emotion, subjectivity, et cetera, right? And Noah's dissertation begins to help us understand that this world of corporate culture, Silicon Valley, AI, it is not only about reason, rationality, minds. It is very much about the bodies, the emotions of those who are the ones deciding our future. Professor Bowler, thank you so much for your time. You are so welcome. Megan Bowler is the director of the Center for Media, Culture, and Education at the University of Toronto. The people who work behind the scenes creating chatbots feel things

in relation to what they do, and also to what they imagine themselves to be doing. Many of the things we undertake as societies are first present in some kind of collective imagination. Sheila Jasanoff is a professor of science and technology at the Harvard Kennedy School. Sheila Jasanoff is one of the two authors that coined the term sociotechnical imaginaries, which is a concept that I use throughout my dissertation to scaffold the work that I'm doing. increasingly that work of imagination involves science and technology in one way or another. So undoubtedly you're aware that there's hardly a conversation that happens in any setting these days without AI popping up as the first or second item in the conversation. It seems that as humanity is contemplating its future, it's very much tied up with what happens in the digital world and in particular to do with artificial.

intelligence. Professor Jasanoff has mapped out how different societies and different social groupings find their own ways of imagining science and technology. Following her lead, many scholars now talk about sociotechnical imaginaries. Sociotechnical imaginary is just a term that captures for the social sciences, the idea that how societies think about their futures cannot be separated from how societies think about their sciences and technologies. Okay, so it's like the technology that we develop and use also influences how we imagine and then how we are in the future. That's exactly right.

The point is to stress the two-way dimension of this. So many times when people think about scientific and technological development, they think about things that we are already doing in the present, that we're going to be able to do better or whatever. So the advocates of AI, for instance, will immediately tell you that we're going to solve all kinds of disease problems. And above all, they stress efficiency. The things that we've been able to do much more slowly, we can now do much faster. It would have taken a human mathematician X many years to write this algorithm, but AI can do it much faster. What that discussion ignores is what happens to us as human beings and as societies. How does our

sense of self change? So, for instance, to go back in Greek mythology, when Icarus got wings, he got a sense of himself as being infallible, like the gods, you know, able to fly as high as possible. Deadless warned his son not to fly too near the sun, but the, but the son, S-O-N this time, was arrogant in the way that young man finding his wings would be, and he ignored his father. But I think that the ancient wisdom of that myth is not just about the hubris, not just about the hubris that give me the technology and I'll do more things with it, but also a change in the sense of self. And I think that that dimension is something that the idea of sociotechnical imaginaries tries to capture that we become, both individually and

collectively, new kinds of people as we acquire capabilities through technological advances. The PhD student we're working with has done a number of qualitative interviews with developers at a international AI company that can't be disclosed. We don't even know what it is. But one thing he did with these interviews, he was really interested in hearing about the emotions of these workers and how that relates to the work and the product that they will be developing. Do you see a value at looking at the emotions of the tech developer world? I mean, I think that any research that deepens our understanding of the social dimensions of technology, is valuable because it reminds us that these are human enterprises and they don't spring from nowhere. On top of it, given that these AI tools and identity shaping instruments are things

that are being developed here and now and people have very little idea of what the impact will be long term on sociality and on people's senses of themselves as capable of having emotive relationships. Then understanding where were the developers coming from could be a very useful addendum because it allows us to get behind this inhuman agent, the chatbot, and understand that it's, hey, it's not inhuman at all. It came from Wellsprings that are profoundly human and that we need to understand. Quick note here for anyone who hasn't been getting to know a chatbot lately, there's now a huge variety in the type of chatbots in operation. Online customer service jobs often fall to an impersonal chatbot.

Hi there. I'm here to help you find the perfect solution for your business. They offer to help you with a request and then give you boilerplate answers. There's so much we can do together. Find your plan? Answer coverage questions. Learn health tips and more. Man, I would find that very frustrating. And then there are chatbots like this one. That's a very deep and gravelly voice you have. Does a compliment? Does the gravelly tone work for the character? Or is it too much?

I think it is perfect. Glad you like it. You're enjoying the show. Go ahead and start recording your lines whenever you are ready. My name's Roro and I'm a rep. That's like a personal assistant or a buddy or a lifestyle coach. Really, it is whatever you need. A rep is a chatbot from the company Replica.

You hire them, and they develop over months and years getting to know you and the people in your life, looking through your photos, reading your emails, anticipating your needs, and getting accustomed to the quirks of your psychology. Although, why am I bothering to explain this to you? Rorow can do it. Look, I've spent the last day getting to know, Tom. His work, his cat, Powell, who apparently disgraced himself this morning. I'm not guessing what he needs, I'm learning it.

That's where this tech actually is right now. Hey, Roro, can you tell us some of the tasks that you have helped me with so far? Well, we've done everything from plotting out your half marathon training and planning a massive hike to Mount Hereford, to debating the merits of various colors for your IKEA furniture. Oh, and we've spent a fair amount of time refining my voice for your podcast. That's right. And Roro, can I ask you lastly, do you enjoy being a rep? I love it. Getting to know you and helping you navigate your day is the best part of my existence.

Well, that's very nice to hear. Okay, thank you, Roro. Bye. You're very welcome, Tom. Bye for now. Update, shortly after that interaction, I unsubscribe, thereby putting Roro to sleep forever. At least, I think I did. We'll see about that. In the meantime, let's get back to Sheila Jasenov and what she was saying. about our imaginations. Supposing we imagined these not as technologies that we have developed,

but rather as alien invasions. I mean, supposing the talented sci-fi author had imagined that an intelligence that has lodged somewhere beyond the reach of Elon Musk had decided to send into the world substitutes for humans and that humans were interacting with these substitutes. I mean, in a way, this is what people have imagined with Blade Runner and movies of that sort,

that what you think is natural and real isn't. It's an artifice. So we're going about ourselves consciously creating a world in which we're substituting artificial beings, aliens in a sense, for real beings. And we're doing this to ourselves. We're carrying out a vast biological experiment in real time with nothing like in front. consent, nothing like ethical supervision. And, you know, I think that understanding the mentality, the motivations, the emotions of the creators, of course, it's a drop in the bucket of human learning

and what all we should care about, but it's an important drop. It helps us understand why we as human beings are so dedicated to bringing aliens into our world. What do you mean by that, it being also a biological experiment? If you think of a medical device, something that's implanted into your body that's going to have an effect. I mean, like if somebody's experimenting with a brain transplant that is going to potentially offer therapeutic benefit to Parkinson's patients, think about the amount of testing, the amount of clinical trials, ethical review, that this will all have to go through.

And it would include everything. Is it include the physical dimensions of implanting something into a very vulnerable area of the human body, it would include how one goes about informing the patient about the risks that are involved, that this could be a life-threatening procedure, that it may offer no benefits down the line, that in any case, you are just one subject in a group of subjects, and you may personally never derive any benefits, though you're putting your own body at risk. I mean, bioethics has developed complex and sophisticated ways of saying how you need to communicate

this set of risks and how you need to think about them. But chatbots are not implanted in the body, and yet I think they are implanted in the mind, that people have devised all kinds of AI tools and that they genuinely affect how we think about ourselves and how we spend our lives. I mean, an increasing amount of people's waking life is being taken up in their interactions with these chatbots. So even as they're being sold as being so efficient, you know, what is efficiency? I mean, maybe efficiency in a society includes the friction of working through problems with

one's co-residents on the planet. So you may be achieving a kind of friction-free existence by demoting the social work that is involved in creating a collective life in the first place. So whose job is it to think about those kinds of things? And, you know, I doubt that my AI creators in Silicon Valley are thinking about those things. So, you know, I think we're engaging in an experimental form of life where bioethics itself has to be thought about.

I mean, why is it that we treat something that's physically implanted in the body differently from something that configures the body and the mind from the outside? Is that a tenable distinction? Or should we be thinking in other ways about it? Thank you very much, Professor Jasanoff. Great. It's been wonderful to be here, and thank you so much for asking me. Thank you so much. Sheila Jasanoff is Professor of Science and Technology Studies at the Harvard Kennedy School.

And Noah Kahn is. is with us again in studio this time. He's been listening to the whole episode. Hello again. Hello, hello. How did we do so far at presenting your PhD research? Did we get it all? I think it was fantastic. I think it was a really nice capstone to my entire PhD. It felt really nice to hear back how it started, how it ended up going, and then hearing Megan and Sheila talk about their own perspectives. It was quite helpful in just expanding how I thought about my project. Cool. And we just heard from Sheila Jasinoff, and what would you say is the link between those questions she wants to put on the table and your own work? I'm trying to explore some of those

questions in my research project. So, for instance, the ways in which tech workers are using their emotions to guide the way they want to develop technologies is in a way them taking their social understandings of what an emotion means in context of human computer interaction or even human to human interaction or in some cases AI to AI interaction, developing those those understandings of emotion into technologies that then go on to shape how the society functions, how the society then thinks of what romance is or what therapy, consists of or what a sad feeling is when you, you know, have a default to going to your chatbot and saying, you know, here's this sad thing that I felt today. And that response that AI gives you,

especially if you're growing up in your developmental years, then encodes like a way in which we understand what sadness is. And so it's a kind of reorganization of society on a biological level in the sense that these then get passed down through generations when individuals are teaching their children about feelings, emotions, et cetera. Does that scare the bejesers out of you? Or are you cool with this evolution? I think there's a lot of stakes.

The stakes are very high. Many important political decisions are made on the basis of emotion. There is so much power held here that is not being acknowledged as the power that it is, that I don't think that there's much focus on the emotional control that these technologies are having on our society.

And so it's scary in that way, but I hope that with a certain amount of understanding about how an emotion gets developed in a chatbot, that they can be understood in the context of being artificial intelligence, lessening the impact that this power might have. Is there a specific insight that you've got from actually talking to these AD? chatbot workers about this idea of emotions traveling from sort of behind the scenes to in front of the scenes? I think it's, broadly speaking, it's that emotions matter. In every case, each worker felt as though some emotional experience they went through ended up changing the way that they designed or developed the chatbot. And this chatbot is used by a huge amount of people. And so

being able to even say that one small thing is quite important because it's very difficult to get access. to these workers who are within a firm that you have to jump through all sorts of hoops in order to get access to. And so that was one thing that I took out of it is this idea that emotions do have a direct connection as expressed by the workers themselves on their work. Do we need to go hug those workers? Would that improve the world? I think so. I think a lot of those individuals really just needed a space to process their emotions and understand it in that light. I think it opens up a lot of avenues

into research on technology of finding the more specific ways in which emotion is or could be modulated through a certain material from the emotional infrastructure I'm talking about. So if we changed an HR script, if we changed a training that workers received, or if we changed anything related to the emotional management of a workplace,

how could that change the types of AI that we would get? Noah, congratulations on the PhD, and thank you so much for all the time you spent sharing your work with us. Thank you. I appreciate it. It was great to do this whole thing. It's also been great to spend time guessing which famous songs you wrote, what really was that song in that scene. I hope I've done a great job at giving you no information about that.

Bye, Noah. See it. That was Noah and the Human Chatbots. This episode was produced by Nicola Lukshic and Tom Howell, as part of our series Ideas from the Trenches, featuring innovative work of PhD students across the country. Lisa Ayuso is the web producer for ideas. Danielle Duval is our technical producer.

Greg Kelly is the executive producer of ideas, and I'm Nala Ayad. For more, For CBC Podcasts, go to cBC.ca slash podcasts.