Daybreak - As the Gen AI race heats up data centres, water becomes the new air

Episode Date: November 18, 2024

Cloud, streaming, generative AI… while all of these are increasingly becoming hot topics of discussion, data centres—the large, boxy buildings that house high-powered computers—are looking for innovative solutions to stay cool.The advent of GPU processing has opened an opportunity for a handful of foreign companies to throw their hat into the ring. Their proposition? Liquid cooling. So far, air cooling has been the preferred way to keep data centres cool. Like its name suggests, it is the process of using air to keep these centres cool. Liquid cooling does just that but with water. It’s more efficient and largely believed to be a better way of cooling. But change does not come easy. Many data centre operators here in India believe it is riskier than air cooling. But with AI technology advancing the way that it is and GPUs growing more popular, they may soon not have a choice. Daybreak is now on WhatsApp at [phone]. Text us and tell us what you thought of the episode!Daybreak is produced from the newsroom of The Ken, India’s first subscriber-only business news platform. Subscribe for more exclusive, deeply-reported, and analytical business stories.

Hi, this is Rohan Dharma Kumar. If you've heard any of the Ken's podcasts, you've probably heard me, my interruptions, my analogies, and my contrarian takes on most topics. And you might rightly be wondering why am I interrupting this episode too. It's for a special announcement. For the last few months, I and Sita Ramon Ganeshan, my colleague and the Ken's deputy editor, have been working on an ambitious new podcast. It's called Intermission.

We want to tell the secret sauce stories of India's greatest companies. Stories of how they were born, how they fought to survive, how they build their organizations and culture, how they manage to innovate and thrive over decades, and most importantly, how they're poised today. To do that, Sita and I have been reading books, poring over reports, going through financial statements, digging up archives, and talking to dozens of people. And if that wasn't enough, we also decided to throw in video into the mix. Yes, you heard that right. Intermission has also had to find its footing in the world of multi-camera shoots in professional studios, laborious editing, and extensive post-production. Sita and I are still reeling from the intensity of our first studio recording.

Intermission launches on March 23rd. To get alert, as soon as we release our first studio. episode, please follow intermission on Spotify and Apple Podcasts or subscribe to the Ken's YouTube channel. You can find all of the links at the ken.com slash I am. With that, back to your episode. Hi, I'm Snigda, the co-host of Daybreak. Before we begin this episode, I have a confession to make. You know, sometimes hosting a podcast can feel like you're talking into the void. You see, we're constantly trying to improve your listening experience. and make this podcast better.

In fact, for those of you who've been listening for a while, you would know that Daybreak Underwent quite a makeover this year. We went from three episodes a week to five. And we also introduced a new Friday special episode. We got a whole bunch of really interesting people on the show. And we even introduced a whole new segment called Daybreak Unwine. But the point of all of this is you, our listener. We want this podcast to be a valuable idea.

addition to your day. But for that to happen, we need to actually hear what you think. And the easiest way to do that is to rate this podcast on whatever streaming platform you use, especially on Apple and Spotify. Leave us a comment, share it with your friends and family. It really, really helps us. And if you have some more feedback for us, you can write to us at podcast at the ken.com or send us a text on WhatsApp. Our number is 89711. 0.8379. I'll say that again. 89711.

08379. We'll also add it to the show notes of this episode. Thank you in advance. And now on to the episode. The last couple of years, all everyone has spoken about is the new technology on the blog. I'm talking about generative AI,

data streaming, cloud services, you get the drift. The thing is, very often when we talk about these incredible advancements in technology, we forget about the very backbone of it all. I'm talking, of course, about the data centres

that house the high-powered computers responsible for this magic. These data centres are essentially large, boxy buildings, nothing much to look at, but they play a key role in the whole AI puzzle. The biggest challenge is keeping them cool. Let me explain. Think about your laptop. If you work on it all day,

it tends to heat up from running at full capacity, right? and then you'll probably hear those little fans inside it whirring full speed to cool it down. Similarly, in these data centres, there's a lot of heat produced by running all of these powerful computers at top efficiency. It's a lot like your laptop, but much, much more intense. In fact, there's a whole industry that's been built around keeping these data centers cool. And now, a handful of foreign players like Vertev, Schneider Electric and Stools are throwing their hats into the ring with a new proper. position. The Ken reporter Abiramiji tells us what's going on. So Schneider, Verte1 strolls have been around for a while in the data center cooling

business. They're pretty big there. And they have traditionally done air cooling, which involves basically what you would expect an air conditioner to do, but for a large set of super high intensity computers that generate a lot of heat. But now their newest proposition is liquid cooling which involves piping water or specialised liquids or you know immersing an entire server into a special kind of liquid in order to cool it directly rather than using air to cool it the reason this sort of cooling is becoming more popular now is because computing is increasingly becoming more and more high intensity and more computing means more heat which makes it all the more important to find an efficient way to keep these computers cool by dissipating that heat.

Like Abidami said, liquid cooling is more efficient. But still, change doesn't come easy. The thing is, operators here in India fear that it is risky. And of course, there are also huge costs involved. But a lot of their big clients, like take phone pay, for instance, have pushed for liquid cooling. The thing is, very soon it may not be a choice, especially now with the global gen AI race in full swing. Welcome to Daybreak, a business podcast from the Ken. I'm your host Rahal Philippos and I'll be joining my colleagues Niktha Sharma every day of the week to bring you one business story that is worth understanding and worth your time. Today is Wednesdays the 19th of November. For decades, the semiconductor industry abided by Moore's law. Every two years, the

latest chip would be half the size of its predecessor with thrice the computing power. But today's scenario is different. Newer generations of computers have exponentially larger computing capacities than their previous iterations. And what that means is that cooling technologies also have to constantly keep updating themselves in order to keep up with the amount of heat that they generate. Cooling is super important, now more than ever. The market is expanding. The market is expanding. at breakneck speed, the world's biggest tech giants, the likes of Amazon, Google, Microsoft are all trying to outdo each other in the GPU compute race.

You can think of GPUs sort of like the CPU we all know, but on steroids. Simply put all the crazy things we've seen AI do in the last couple of years, like create vivid images out of nowhere, throw up crazy human responses to complex questions and process massive volumes of data, all of that is largely possible thanks to GPUs. And it isn't just the big guys. A lot of enterprises are looking to get in on the AI gold rush. And this, of course, demands more GPUs. Abirami spoke to eight different people

and all of them said in this scenario, liquid cooling is the answer. You see, GPUs are great and all. They pack several times the computing power at a fraction of the space taken by traditional CPUs. But it comes at a cost. Much higher power use. A key metric here is,

is the power usage effectiveness or PUE. Now, this measures how efficiently a data center uses power. A completely efficient system would have a PUE of 1, but in practical terms, this metric is usually between 1 and 2. Below the PUE, the more efficient the data center is. Now, here in India, most data centers have a PUE of 1.6 to 1.8. Take InfoSys for instance. It has a PUE of 1.55.

One expert Abidami spoke to said it was because ambient weather and temperature conditions in India drive up the energy required for cooling. That's definitely one reason, but they also said that the other reason was because they haven't invested in the most sustainable measures available in the market. Liquid cooling seems to be that option. Data centers that use air cooling to lower temperatures typically devote half of their energy usage to the setup and other overheads. Compare that to facilities that use liquid cooling. where only one-third or one-quarter of energy usage is devoted to cooling. But of course, like with most things, it isn't quite as straightforward as ditching air cooling

and replacing it with liquid cooling. Stay tuned. Let's talk about how much energy a data center consumes. But to make it a little more fun, let's compare it to the average energy consumption of a household in Bangalore. Okay, so a house with about 2 to 4 residents would use around 0.81 megawatt hour of energy for the entire year of 2023. Compare that to a data center.

Typically, it would use 50 to 100 megawatt hours in one single hour. Now, that's enough to power as many as 120 households in a year. Again, air cooling for racks below 50 kilowatt require larger, bulkier coils that take up a lot of space. For context, racks are those metal frames used in data centers for organizing IT equipment like servers and switches. Now, when air cooling comes in and energy goes up, it requires larger, bulkier coils,

which obviously takes up a lot of space and that in turn drives up the cost and the amount of power utilized. Technically, water cooling is a better way to go. But it's difficult to implement this kind of tech, hence the low adoption rates. For instance, of its total data center capacity of around 380 megawatt, a major provider like NTT uses liquid cooling to cool only about 6.4 megawatt.

And that's because of technical challenges like condensation, possible leakage, and laying pipes that can withstand high temperatures. Now, these are all unique challenges that come with liquid cooling. But things are changing, attitudes are changing. Companies still care about sustainable practices and clients have been expressing their interest in implementing solutions like liquid cooling to improve their sustainability footprint. Now, that's exactly why the likes of PhonePay

had instructed operators like NTT to install both varieties of liquid cooling, direct-to-chip and liquid immersion methods in its data center servers. But operators, including NTT, are very about data safety and power costs. NTT's first reaction to phone pay's request was to make it clear

that they would not take responsibility for a liquid-cool system. So what happens then? Stay tuned. The reason operators are concerned about liquid cooling is because it's a completely different ballgame. It's riskier and makes data centers liable if their servers malfunction.

Now, compared to that, air cool systems are less risky. They can't cause any kind of damage to the servers because they don't interfere with their functioning at all. The other concern with liquid cooling is the high up-front cost involved. In fact, it can cost up to three times the money needed to implement an air cooling. system. But on the flip side, in terms of overall power saving, this cost is likely to be recouped in the next three to four years, which is why the tide is turning in favour of liquid cooling, in the

international market at least. Take stools, for instance, they've been seeing inquiries from countries like Saudi Arabia for liquid cooling data centres that use tens of gigawatts of power. People in the space say that once the shift to liquid immersion cooling becomes mainstream, there'll be a drastic dip in demand for any other kind of cooling. And now that Nvidia chips are entering the Indian market, power consumption is only going to shoot up further. We are looking at a near future, where power consumption may not even be measured in kilowatts or megawatts anymore.

It'll be measured in gigawatts. That's the amount of power that these chips will require. And cooling technology will have to keep up. Daybreak is produced from the newsroom of the Ken India's first subscriber-focused business news platform. What you're listening to is just a small sample of our subscriber-only offerings. A full subscription unlocks daily long-form feature stories, newsletters and podcast extras. Head to the ken.com and click on the red subscribe button on the top of the website.

Today's episode was hosted by Rahil Filippo's, produced by me Snigda Sharma and edited by Rajiv Sien.