In The Arena by TechArena - Inventec Delivers Cutting-Edge Cooling for AI Workloads

Episode Date: October 22, 2024

In this episode, Inventec’s Edward King discusses how AI workloads are shifting to the edge, and how innovations like liquid cooling and ruggedized infrastructure can power the next wave of tech advancements.

Welcome to the Tech Arena, featuring authentic discussions between tech's leading innovators and our host, Alison Klein. Now, let's step into the arena. Welcome in the arena. My name is Alison Klein. We are coming to you from the OCP Summit in San Jose, California this week, and I'm delighted to be joined by Edward Ting, VP of New Ventures at Invent Tech. Welcome to the program, Edward. Hi, thanks for having me. And this is the first time you've been on the show, so why don't you go ahead and just introduce

yourself and give us a little bit of information about InvenTech. Yeah, I've been in the industry for quite a while. I've worked at InvenTech for the last 21 years. I started in InvenTech doing product development for them with a variety of enterprise and consumer products. Found my way to strategic supply chain development for InvenTech. And most recently, I run business development for our emerging technologies in North America. Now, when we think about InvenTech, what kind of technologies should we think about in that scope of focus? InvenTech is quite a unique company. We're an $18 billion company that no one knows who we are. We're the brand behind the brand, and that's always been. Our strategy is to partner with the large people in the industry to bring solutions to market.

So when you think of InvenTech, you can think of us as the company that produces everything from AI racks to wearables to hearables to personal compute and everything in between. That's an incredible purview. You're focused here at the show on talking about infrastructure in the AI era. You're focusing on edge servers and liquid cooling. We are at this show.

And tell me about what your talk is talking about and why you chose those topics for this moment. OCP and the concepts behind OCP are important to us in our design phase. Because we put together solutions for a lot of our customers, the modularity helps us to quickly put together solutions. But part of the quickly putting together solutions are the interconnects that happen between that. And the one that we find recently that helps us the most is with liquid cooling and cooling, especially with our

large data center CSP customers. So this has been part of our purview at this show is to highlight that fact. And then for Edge, we have been with the edge for quite a while now. And we see this as a growing part of the overall infrastructure is to move compute and analytics to the edge where the data is generated. And so it's been growing for us for the last seven years and will continue to grow for us. Now, obviously, AI has been a huge focus of the OCP community, a lot of focus on AI training in the data center. You're focusing on edge, and I'm wondering if you see a transition of AI workloads being primarily focused in the data center to growing out to the edge as we move more towards enterprise applications of inference of

the edge. Yeah, we see that all the time. We work with several of the inference companies, both in the large language model space and on the edge with maybe small language models and other neural nets on the edge. We see a tremendous volume now, especially in the video space, of video analytics moving to the edge because of the sheer bandwidth required to move video data across the net.

A lot of companies have decided to move the analytics and the video itself and keep it on the edge. Now, when you look at that proliferation at the edge, what's the difference in infrastructure for those edge environments versus what's the difference in infrastructure for those edge environments versus what's being maintained in data centers today? A lot of data centers are very hygienic, right? Clean, air-conditioned, but the edge is not. And so a lot of the challenge on the edge becomes in ruggedizing these solutions that go out on the edge because they live in places that can be hot, can be dusty, can be very humid. And so a lot of what we do is to help not only move the analytics

to that edge, but also the infrastructure that needs to move there has to be more rugged than a data center. And the power requirements are much different at the edge. The power, it can be a steady power or it can be power that's not so steady. Somewhat erratic, I would think. Very erratic at times. Interesting. Now, let's go back to the data center and liquid cooling for a second. Obviously, compute is getting much more dense with AI workloads. Tell me a little bit about how you're seeing cooling evolve. And there are different types of cooling out there. There's still not a confluence of thinking regarding direct cooling versus immersion cooling. What do you see the trend in the hyperscaler space and

how do you see that proliferating into the broad market? We have seen cases where immersion cooling and direct liquid cooling are both applicable in the data center, but we feel to scale these data center, there needs to be perhaps a hybrid approach so that with the immersion cooling, you can get good PUEs with immersion cooling. But our experience is they're not always easy to use and they're not always easy to maintain and they're not very serviceable. Direct liquid cooling, however, you can cool certain components and achieve good cooling in the infrastructure this way. But it doesn't scale, I think, as well as the immersion cooling does.

So we have now taken a hybrid approach and come out with a product we call Lambert that is a rack-mounted node that encapsulates the processing unit. So you get the best of immersion in this sealed node. And in addition, we run direct liquid cooling in these nodes to the silicon that can be the hottest. We truly are trying to find the best of both worlds in this cooling. And because it's rack mounted, think of a beehive and being able to pull in and out the sections of the beehive. It's much like this. It will be in a normal rack, but it's easier to scale. It's less messy than dealing with immersion and immersion tanks.

And it's easy to replace nodes. Rather than taking the whole tank offline, you just have to pull out a node and then replace it. And as you replace it, it syncs up with the cooling in the rack, and it also syncs up with power and the networking in the rack. That's fantastic. Now, OCP Summit has always been an event where you can see the leading edge of technology. What hyperscalers are thinking about in terms of what they need next and how the industry is going to respond to that? What topics have really piqued your interest that you've heard at the show? And is there anything beyond the topics we've discussed so far that have really got your attention? We talk a lot about heat and power racks, but the thing that I've been tracking for a while is CXL. It's not only the heat and the power, but the ability to efficiently move bits around inside the units itself. That is something that's piqued my interest this year. And we've seen more and

more solutions being rolled out for CXL. We're rolling out CXL solutions on the edge so that you can have maybe a lot of GPU power on the edge. You may not need such a high powered CPU because CXL allows the GPUs to get to the system memory. And it allows this throughput to be better. And so CXL is something that has piqued my interest for the last probably year. And I've seen more and more memory companies come online with CXL designs. So are you seeing it in implementations for shared memory spaces? Are you seeing it as an inter-process or interconnect? How are you seeing CXL actually manifest in these designs?

In both places. It's the memory one that I think has come online first. The interconnect, I think, is the thing that's being exposed now that I find really fascinating and interesting. I think will push the market forward a little more. It's so interesting that you say that, because we were talking all about CXL a year ago, two years ago, and it almost feels like people are getting quieter as people are implementing it in real world solutions. This is it, right? And, you know, it's the same with AI.

I think we're at the top of the hype cycle wave just like we were with CXL a couple of years ago. And maybe next year, this show and other shows, AI won't be everywhere, but it'll just be quietly being implemented. And I think CXL is this way now. It's a great example of an industry standard to build a top PCI Express

and something where the industry has really coalesced around a solution. OCHE is a great place to track standards progress. Last year's show, there was a ton of conversations about ultra-Ethernet. Is there anything else at the show that has grabbed your attention? I think this year, there are more people here than there have been, which is exciting for the industry in itself. As you look at the economies around the world, there's a lot of people that are taking kind of a weight because of the U.S. election.

But seeing the number of people who have shown up here who are truly enthusiastic about pushing OCP forward, the solutions forward, it's very heartening to see this. Yeah, no kidding. I mean, I think that it's fantastic. It's the largest OCP summit in the history. It is. And the amount of lean forward conversations that are happening, real conversations about pushing designs forward rather than just showing up in another trade show. Yeah.

It's nice to see. And I sensed that at the opening reception last night and all the conversations today at Pitch of Art. Yeah, much more enthusiastic than it has been in prior years. I agree with that. Now, you talk about InvenTech being the company behind the company. One question that I have for you, and it's probably my final question, is I'm sure there are people that are listening. They're like, I want to find out about what InvenTech is making because you are the company behind the company. Where would folks find out more about the solutions that you talked about today and other solutions that you're delivering to market and maybe connect with you. So you can start with our website, www.invented.com. And if something

there piques your interest or is something that you've heard today or in other conferences, you can always reach out to me on LinkedIn, or you can find me at king.edward at invented.com. King Edward. You got the best email ever. Thank you so much for being on the show. Congratulations on your session today. It was great catching up. Thank you. Thanks for joining the Tech Arena.

Subscribe and engage at our website, thetecharena.net. All content is copyright by The Tech Arena.