Daybreak - Is there room for deep fakes in democracy? AI startups seem to think so

Episode Date: April 16, 2024

Just like every Lok Sabha election in the last 72 years, millions of people will vote for a new government over the next couple of weeks.  But there is one thing that really sets this election apart. Never before have political parties actively used Generative Artificial IntelIigence at this scale. It is a turning point in India’s electoral evolution. Some AI startups in India have been developing hyper-personalised voter experiences for political parties. This comes at a time when Gen AI tools like deepfakes have become very sophisticated — to the point where even experts often  struggle to tell what is real and what is not. In the run-up to the election, when you are being bombarded with political content, videos and images, this can be very dangerous. Yet, there are barely any rules in place to regulate the use of this technology during the election process. What does this mean for the world’s largest democracy? Daybreak is produced from the newsroom of The Ken, India’s first subscriber-only business news platform. Subscribe for more exclusive, deeply-reported, and analytical business stories.

Hi, this is Rohan Dharma Kumar. If you've heard any of the Ken's podcasts, you've probably heard me, my interruptions, my analogies, and my contrarian takes on most topics. And you might rightly be wondering why am I interrupting this episode too. It's for a special announcement. For the last few months, I and Sita Raman Ganeshan, my colleague and the Ken's deputy editor, have been working on an ambitious new podcast. It's called Intermission.

We want to tell the secret sauce stories of India's greatest companies. Stories of how they were born, how they fought to survive, how they build their organizations and culture, how they manage to innovate and thrive over decades, and most importantly, how they're poised today. To do that, Sita and I have been reading books, poring over reports, going through financial statements, digging up archives, and talking to dozens of people. And if that wasn't enough, we also decided to throw in video into the mix. Yes, you heard that right. Intermission has also had to find its footing in the world of multi-camera shoots in professional studios, laborious editing, and extensive post-production. Sita and I are still reeling from the intensity of our first studio recording.

Intermission launches on March 23rd. To get alert, as soon as we release our first video. episode, please follow intermission on Spotify and Apple Podcasts or subscribe to the Ken's YouTube channel. You can find all of the links at the ken.com slash I am. With that, back to your episode. Over the next couple of weeks, millions of people across India will vote for the country's next government. In all likelihood, you are one of them. I don't know if you've thought about it, but there is one thing that really sets this election apart. I'm talking about generative artificial intelligence.

Never before have political parties actively use Gen AI at this scale. It's almost like a turning point in India's electoral evolution. I want you to picture this. You receive a personalized video from a big-time political leader. Like, say the Prime Minister, he's addressing you by name. He somehow knows what you do, what your interests are, and what your voting history has been like so far. He's appealing to you to vote for his party, telling you why it is the right choice for you.

Now, imagine a similar video going to lacks of voters across the country, all tailor-made for them. It sort of sounds like something straight out of a black-mirror episode, but what if I told you, some AI startups in India have been developing hyper-personalized voter experiences just like this one. their clients, political parties of course. In the last year or so, Gen AI tools like deepfakes have gotten really, really good to the point where even an expert would struggle to tell if they're real or not, which is why when it comes to AI,

the line between what's real and what's not can get blurry very fast. And in the run-up to an election, when you're being bombarded with political content, videos, images, it can be dangerous. I'm talking misinformation, hate speech, political propaganda. You know that's a deadly combination right before a big election. And yet, there are barely any rules in place to regulate this technology. What does this mean for the world's largest democracy?

Welcome to Daybreak, a business podcast from the Ken. I'm your new host, Rahal Philipos, and I will be joining Snigda every week to bring you one business story that is worth understanding and worth your time. Today is Tuesday, the 16th of April. Hello. Barack Obama's AI Avatar. Yes, yes.

Let's say, tell you. What you just listened to was a tiny sample of how generative artificial intelligence could potentially change the very nature of political campaigning in the country. What I played for you

was a snippet of a conversation between Deweindar Singh Jadu, the founder of an AI startup called the Indian Deepfaker, and his own brainchild. AI Obama. You see, AI Obama is not your average robocaller. None of its responses are pre-recorded. It's able to have unique one-on-one conversations with a potential voter. Listen to how AI Obama articulates its priorities for the election and then ask Jadu what he thinks. I think it's a

idea that your area have been bidgian please our problems to give us so we can

can't and you can how we can't our party political leader's voice. That's the power of generative AI. Like the Indian Deepfaker, other AI firms in India see the election as a great testing ground for new technology. And two-way calls are just the tip of the iceberg. These AI startups are constantly innovating. Some of them are even toying with the idea of building large language models or LLMs like

Chat GPT, trained on leader's speeches. It's sort of like Neta GPT. Imagine a personalised chatbot of Rahul Gandhi. It would talk to you, answer your questions in his signature style. But the AI tool that political parties just can't seem to get enough of is deepfakes. Parties have been creating and sharing deepfakes more than ever before. More on that in the next segment. These AI-generated hyper-realistic videos have a bit of a bad reputation. And rightfully so.

Remember that deep-fake video of actress Rashmika. Mandana that did the rounds on social media a while ago, her face was superimposed onto a video of British Indian influencer Zara Patel. Most people who saw the video fell for it. It was only later that people noticed some minor glitches here and there. But one good thing that came out of the incident was that it got people talking about the dangers of deepfakes. Think about it in the context of an election.

A deepfake made with malicious intent could potentially damage the credibility of a political leader or a party. It could also completely change your perception as a voter. Or it could also do the opposite. In January this year, five-time Tamil Nadu chief minister and political icon M Karunanadi made a surprise appearance at a DMK event. But Karunadiki has been dead for about six years now. An AI clone of the late DMK leader delivered an eight-minute speech at the event. Right from the million million dollars million dollars the result of

the students for the value money-pourer. Right from Karunanadi's voice to his signature black glasses and yellow shawl, the resemblance was uncanny and all his supporters loved it. So much so that AI Karunanadi

was brought out on multiple occasions after that. This particular AI clone was developed by Senthil Nayagam, the founder of Moonium AI. When Ken reporter Abhirami spoke to him, he said that it didn't take much to pull off a stunt like this. It could take as little as a voice sample and two photos. With generative AI becoming a lot more accessible, it doesn't take an expert like Senthil to make a hyper-realistic deep fake anymore. Anyone can create one, including the average party work. or a miscreant looking to cause trouble. There are so many easily available software and apps for that sort of thing.

And for the slightly more complicated projects, there's always freelancers. And that is when things can get even more messy. I'm sure you've come across a few of these videos yourself, like right before the Telangana elections, when a doctored video of Bharatrashrish Samhiti leader Katie Rama-Rao urging people to vote for the rival Congress Party did the rounds on social media. Here's the clincher. The fake video was posted on the official social media handles of the Telangana Congress. And this wasn't just a one-off. There have been multiple instances where political parties have shared similar fake videos.

So you can see why monitoring and controlling deepfakes is a huge challenge for social media and messaging platforms like Facebook and WhatsApp, which brings me back to what I was saying at the beginning of this episode. when you are bombarded with videos and images during elections, it's hard to tell what's real and what's not. Think about the average voter in a remote part of the country. What happens when they encounter a hyper-realistic deepfake propagated by a political party? Or an AI-trained chatbot of a political leader

with its own inherent bias. It could be a real can of worms. And yet, the model code of conduct, which is sort of like an instruction manual for, political parties and their campaigns, did not mention AI anywhere. In fact, the Indian government seems to be tiptoeing around AI regulations. Stay tuned. In March, the Ministry of Electronics and Information Technology asked tech companies to seek

its permission before launching what it described as unreliable or under-tested generative AI models. Their fear was exactly what we've been discussing all this while. What does Gen AI mean for Indian democracy? Could it potentially pose a threat to electoral integrity? But people in the AI space found Métis directives pretty vague. It's also still unclear whether regulatory bodies like the Election Commission of India are fully aware of the extent to which AI is being used during the elections.

Most startups dabbling in the political campaigning space have been stressing on the importance of keeping things kosher, during the elections. Remember, Cental Nayagam, the brains behind AI Karunanadi? He even floated something called the Ethical AI Coalition Manifesto, which a bunch of AI start-ups have already signed. Here's my co-host, Snickda, reading out a snippet from the manifesto. We pledge to uphold the integrity of democratic processes

by ensuring that AI technologies are not used to manipulate elections, spread misinformation, or undermine public trust, in political institutions. AI tools deployed in the political arena must be transparent, accountable and free from bias. A similar set of guidelines was signed this year by 20 big tech companies

including meta, Google and OpenAI. They called it the tech accord. They promised to work together to regulate AI manipulated videos, audio and images across platforms on a global level. Are these a step in the right direction? Yes.

But are they enough? Most certainly not. You see, this isn't just a big year for democracy in India. It's election season in several countries across the world. And there's a collective fear around generative AI. In the US, many states like California and Texas have passed laws restricting the use of artificial intelligence in political campaigns. In fact, lawmakers in some of these states are pushing for a full ban on AI in political communication. The European Union has also passed laws regulating AI ahead of the elections.

It requires AI companies to ensure their products meet certain transparency standards. But however you choose to deal with it, one thing is clear. Generative AI is here to stay. As the tech gets better and more accessible, who knows what India's next general election will even look like. Hopefully by then, the election commission will have, some solid guidelines and regulations in place. Daybreak is produced from the newsroom of the Ken, India's first subscriber-focused business news platform. What you're listening to is just a small sample of our subscriber-only offerings.

A full subscription unlocks daily long-form feature stories, newsletters and podcast extras. Head to the ken.com and click on the red subscribe button on the top of the website. Today's episode was hosted by Rahil Filippo's produced by me, Snigda Sharmat. and edited by Rajiv Sien.