In The Arena by TechArena - Insights on AI & Sustainability from Jonathan Koomey

Episode Date: October 23, 2024

Jonathan Koomey of Koomey Analytics shares insights on AI’s role in energy efficiency, sustainability, and the tech sector’s potential to address climate challenges in this must-listen episode.

Welcome to the Tech Arena, featuring authentic discussions between tech's leading innovators and our host, Alison Klein. Now, let's step into the arena. Welcome to the arena. My name is Alison Klein. We're coming to you from OCP Summit in San Jose, and I am so excited for this podcast. I've got Jonathan Coomey, president and founder of Coomey Analytics with me. And Jonathan, your background goes back across so many different

places. Welcome back to the show. And why don't you just start with an introduction of yourself for those in the audience that haven't caught your previous episode. Allison, thanks so much for having me on again. I started work at Lawrence Berkeley Lab in the 1980s. Some of my early work was on the energy efficiency of computing. And that work continued for many years and around about the time that the dot-com boom was happening, there was a couple of guys running around talking about how the internet was going to use half of all our electric power. That was the beginning of a real deep dive into claims that people make about computing and electricity. So my research group at Lawrence Worthy Lab dug into those claims and found that

there's no way that electricity use of IT was going to be 50% and that they were exaggerating in many ways. So I've spent a career, part of my career at least, understanding trends in computing technology and how those trends affect energy use and efficiency. And that's really one of my primary areas of focus. And that's one of the things that we've talked about over the years in various venues. Now, I'm going to get and dig into that, but I wanted to talk to you briefly. talking about your book that you and Ian Monroe published about the climate crisis and what leaders from across sectors could do to help address it with pragmatic approaches. I wanted to ask you what the response has been to the book and what you've seen from that in terms of

any changes in attitudes from business leaders on this topic. The book is called Solving Climate Change, A Guide for Warmers and Leaders. And it's a textbook. So our goal was to make a resource for university professors and others who are teaching about climate solutions. We taught a class a couple of times at Stanford on this topic. And we decided that it would make sense to boil down what we've learned and actually expand what we know. Because the process of writing a book gets you to think anew about things that you hadn't fully thought through. So the book itself, I think, is important because it takes a more comprehensive approach than many other books of his kind. There are a lot of people who are CO2-centric,

so they're focused on carbon dioxide. There are people who are focused on a particular sector, say industry or something else. There's people who focus on other gases than CO2. We wanted to give a high-level view across all the sectors and to include not just technical solutions, but also policy solutions. So we have eight pillars of climate solutions and five of them are technical, but we also have a pillar which is elevate truth. Because if you're going to talk about these issues,

you have to make sure you're talking about facts. So there's that. And then there's mobilized money. And then there's institutional change and there's other. But we wanted to have that more comprehensive view that wasn't just technology focused. It wasn't just CO2 focused. It actually looked at the whole range of actors in the society and the range of opportunities that exist to reduce emissions throughout the economy. I read the book in advance of the first interview, and I know that you say that it's for university professors and classes, but I think that anybody that is interested in this important topic should pick up a copy and take a look at it. Thank you. We tried to make it readable. I think both Ian and I come from a community of practice. Yeah. We want to make a difference in the world. So we didn't want to make something that was dry, but it does have a lot of references.

Yeah. So the professors are very happy with it. Yeah, exactly. It's well-researched. Now, let's get into the topic of the day, which is really around the compute sector. I feel like we're in the midst of kind of an existential moment. We have an incredible opportunity for technology to contribute to some of the world's largest challenges with the advent of AI. And we see DeepMind coming out with the first Nobel Prize, arguably. It is AI-inspired for their protein

research. But at the same time, climate impact is growing at an alarming rate. The impact of? Climate change. Okay, just in general, not just the tech sector. No, just in general. We see these huge move in technology, but also we're really starting to understand the impacts of climate change. And I think that some of us have been studying this for a long time.

Obviously, you are somebody that has been watching it for a long time. But I think that general population is starting to see it's getting warmer. These weather events and these fires and all of the things that are happening, there is truth to the fact that the planet is changing. How do you see this landscape from a standpoint of what's happening with climate and what's happening with tech? The most important thing to understand about climate is that the effects of greenhouse gases, of emitting greenhouse gases like carbon dioxide, are cumulative. And CO2 stays in the atmosphere for centuries.

So once it's there, it's pretty much permanent. And over the centuries, it continues to warm the Earth. It's like a blanket that keeps the Earth warmer and warmer as you have more CO2 in the atmosphere. One of the most important lessons from that is that we need to get to zero emissions as quickly as we can. And most people think with many problems, if you wait a little bit, it's not so bad. You wait and see. But the problem with a cumulative issue like this is that wait and see actually works against. And right now we're seeing the cumulative effects of the last century and a half of

significant greenhouse gas emissions with more intense rainstorms, more intense droughts, more hurricanes, more intense hurricanes. And we're seeing these effects on people's lives before most of the scientists thought that they would happen, at least to this extent. So I think we're pretty much at this crisis point where people need to realize it's time we have to actually do something. And we have to move much more quickly than anyone had anticipated. When I started studying climate solutions in the late 80s, we had time to make gradual changes and solve the problem, but we are out of time. Now we need to treat it like the emergency that it is, and we need to start reducing emissions as quickly as we can, to get to zero as quickly as we can. Now, take a look at the tech sector. You've always been a proponent that the tech sector can help with solutions in this space. We're here at OpenCompute, the land of the hyperscalers.

They have incredible power in terms of their compute capacity. What is the opportunity statement from the tech sector to take action to help with this? I think of information technology as our ace in the hole. It helps us to solve societal problems in many ways. It also, to be fair, can cause societal problems, and we can talk about that afterwards. But the kinds of things that tech helps with turn out to be very important if you

want to decarbonize or reduce emissions in the system. So it helps us with collecting data. It helps us with responding to data in real time. It helps us substitute smarts for parts so we can avoid materials. We can move bits instead of moving atoms like we do with cloud computing to save energy. And we can use it for design. So there are many dimensions in which technology can help get to a sustainable society, a zero emission society. But the tech has to be applied in the right way. Just like you can use cloud computing to make your business a lot more

efficient, you could also, if you're an oil company, use cloud computing to make your business a lot more efficient. And so there are often trade-offs in these kind of situations. I'm optimistic that tech can still be a force for good and can help us get to a zero-emission society more quickly than we could otherwise. Now, the tech sector and data centers in particular are getting a lot of attention for the rising amount of energy that is being consumed to fuel all these computers. OCP has been a huge voice in driving more attention to sustainability, and you can argue the reasons for that. Hyperscalers know how much energy they're consuming and how much carbon emissions that they're responsible for,

for their data centers. But how do you see the position of data centers today? And do you believe the hype that data centers are going to rise and consume up to, depending on the study, 17, 20% of energy in the years ahead. Sounds a lot like what people were saying in the late 90s. So for context, the best recent study of which I'm aware was for 2020, and it showed data centers used about 1% of the world's electricity. AI data centers at that time, relatively small. Maybe they were 5% or 10% of that. So 0.1%. We certainly think that there has been growth in the AI sector in particular. The hyperscalers have documented pretty significant increases in their own electricity use. What is not clear is what's being displaced. So what happens is, and this happens historically for years and it continues to happen, the

hyperscalers are very much more efficient than the corporate data centers. And so a lot of the corporate data centers have been displaced by hyperscale compute. Even though there is a growth in hyperscalers, it actually led to significant reductions in overall energy use. And so we saw from 2010 to 2018, an increase of about sixfold in compute output, but electricity use went up 6% over that period. And we think that there's now growth again, like it was flat for a while. We think that there's now some growth. What happened in the early 2000s was a similar period of very rapid growth.

We had a doubling of compute electricity use from 2020 to 2005. And then the industry really focused on it. And you saw the growth start to level off. And then by 2010, it flattened out. And I expect a similar sort of pattern here, is that in the initial phases, people are buying as many H100 nodes as they can get their hands on and putting them in without regard for infrastructure constraints or anything else. But now people are realizing,

oh, we're seeing some constraints. So how fast can we build data centers? How fast can we deploy H100 nodes in our existing facilities? What do we have to change to make that happen? Oh, do we need to make things more efficient? Those are the kinds of constraints that lead the industry to change what they're doing and become a whole lot more efficient. OCP is also expanding the conversation from just pure energy consumption into embedded carbon, water usage, and true circularity, looking at modular designs, which I love. Governments and EU are requiring reporting at a level that we haven't seen before. Do you see this becoming

a global requirement? And do you see the standards coming from OCP and the work from the EU flowing into how all data centers will be managed? I can't speak to what policymakers will do. I do know that the concern over emissions, embedded emissions, as well as operational emissions, is only going to go. And so I think there will be continued focus on this. I think OCP has an important role in defining standards of measurement and standards of performance for different kinds of equipment. One of the constraints that I have in the research that I do is data. And a lot of times the data just are not available. And usually the data are known internally to the companies, but they don't release it. So Eric Massonet and Noah Lay and I wrote a paper recently that talked about the need for a data revolution. But we really do need to change our practices about data in order for analysts to be able to keep up because we're always behind, right? Because the industry moves so fast and then there's a lag to

release data and then it takes time to do analysis. And so by the time you've done your analysis, the industry has moved on. So there are ways that one could imagine to release data that doesn't cause proprietary concerns. There are industries that have made institutional arrangements to allow so the cement industry, for example, they have this effort where all the players in the industry contribute data to a central nonprofit that they have set up. And that nonprofit releases aggregate data and doesn't release anything that would allow identifying information. And so you could imagine ways to do similar sorts of things in the tech sector. I could see OCP being involved in that sort of... Arbitrator of shares.

Yeah, exactly. So you find an independent group who's very serious about protecting proprietary information, but also wants people to have up-to-date information about what's happening. And so that's something we, as an industry, I think the industry needs to think about. And we need to have conversations about that with us in the research community and OCK and others. Hyperscalers have a history of avoiding a lot of their carbon impact by investing in alternative energy sources and carbon credits. And now we're seeing investment in nuclear energy to power off-grid. How should we look at this in terms of things like the reopening of Riyadh Mile Isle and from a

sustainability perspective? My view is that we need zero emissions power. And if there's a way to keep an existing nuclear plant open when it would otherwise close or reopen an existing nuclear plant, we should do that. If we can do it safely, and we pretty much can. The question about whether new nuclear will be a factor is ultimately a question for the nuclear industry itself. Because the myth is that environmentalists and regulators destroy the nuclear industry, but the reality is that the nuclear industry can't build plants on time and on budget. The costs and delivery time of a new nuclear reactor is indeterminate in the United States. No one can or will tell me what that number is. I can go to a solar developer

and say, I would like to have this much solar on this site. What can you do for me? And they'll tell you, this is how long it takes and this is what it will cost. And there's always contingencies, plus or minus 10%. But no one will give you a fixed price for a nuclear plant. People will give you a fixed price for a solar plant. So I would applaud greatly if the nuclear industry was able to do that. They're not now able to do that, at least in the US. I think in South Korea and in China, they've clearly been able to build reactors faster and presumably more economically, although those data, to my knowledge, have not been independently audited. So we've done analysis of the historical costs of reactors, and the utility says one thing, you have to make sure. So Jonathan, when you look at AI data centers

and the leading edge, you see the amount of power that each of these rocks is drawing. What do you make of this? And how hyped is the new energy crisis of data center compute in the lens of AI? First, let's start at the highest level in the United States.

If you look at the electricity use in the United States, it was lower in 2023 than in 2022. So there's no evidence at the aggregate level of massive growth. You can see in some states that there has been growth. So Virginia is the prime example. Right. Right. There are a lot of data centers. Tons of data centers going into the East.

So that's it. But it's important not to assume that just because it's happening in Virginia, that it's happening everywhere. And so I think that's one of the problems that come into these discussions is that they look at one example and then decide that it's happening everywhere when it isn't. Now, the data are still coming in. We know that there are a lot of H100 nodes, for example, that are being shipped, millions and millions. There are people doing the kind of crude analyses that you've seen by McKinsey and others, and they're making some rookie mistakes. And here's a rookie mistake for you. So NVIDIA, if you look on the website for an H100 node, it says it uses 10.2 kilowatts. Well, that is the rated power

of that node. The rated power is the size of the power supply. And there are still people who believe that is how much power an H100 node uses. And I have been working with folks at Brookhaven National Lab and looking at other measurements. The maximum that they've been able to get these things to use is about 8.5 kilowatts. They're always going to over-provision. And many of the workloads, actually, it's more like five kilowatts. It's these kind of details that actually are important for really understanding what's going on. And a lot of the people who rush in to make these wild estimates are making rookie mistakes like that. And so it makes me very

skeptical. And I've seen this so many times. The investment banks do this all the time. They did it in the late 90s with the dot-com stuff, and they're doing it again. Now, it doesn't mean there won't be growth in electricity use in some areas because of AI and because of data centers. But I think we need to be very grounded in the data to make sure that we really understand what's going on. You want this to actually be based on data. That's shocking. That's just my bias. So here's another example of something that happens in these estimates. When data center companies shop around for power, they go to multiple utilities and they say, we're thinking of a data center that looks like this. They want to know what will be the price of the electricity and demand charges and other stuff. They do that for

multiple jurisdictions, but they don't build data centers in all those jurisdictions. So there's a real danger of double counting. Anecdotally, there is double counting going on. I think the utilities are starting to get smart to this, but that's another case where a lot of utilities are reporting interest in data centers, but not all of them are going to get smart to this, but that's another case where a lot of utilities are reporting interest in data centers, but not all of them are going to get actual data centers. Right. So we need to be very careful. We actually pulled the historical data and the utility forecasting data from recorders of the utilities in the US. And what we see is that the forecasts

before 2022 almost always overestimated demand. Like they would make a forecast that would show growth and then electricity use would be flat. And again, it happened for a long time. In 2022, the forecast went up a little bit and then the demand went up a little bit. 2023, the forecasts are huge for many of these service territories. But whether that actually happens or not, that tells you what utilities expect. That doesn't mean that it will actually. So we need to be very careful in how we understand the forecast that utilities are making. Those are not reality, at least not yet. Okay. Now, final question for you. I could talk all afternoon with you, but we're running out of time. I'm sure that people who are listening

to you want to engage more and want to continue the dialogue with you. Where should they reach out to you, Jonathan? So my website, kumi.com, K-M-O-M-E-Y.com is where I post my blog posts. When I do reports, I post from there. My contact information is there and I welcome contacts from people anywhere because I almost always learn something. Awesome. Thank you so much for being on the show today. My pleasure. Thanks for joining the Tech Arena. Subscribe and engage at our website, thetecharena.net. All content is copyright by The Tech Arena.