@HPC Podcast Archives - OrionX.net - @HPCpodcast-90: Lenovo on Memory-Centric Architectures – Industry View

Episode Date: September 25, 2024

In this episode of the @HPCpodcasts Industry View we are joined by Patrick Caporale of Lenovo to discuss memory-centric architectures, Non Von-Neumann architectures, advances in I/O, CPU-Memory-I/O gap, computational storage, NUMA, disk, tape, Ultra Ethernet Consortium, Ultra Accelerator Link, CXL, tiered storage, power and cooling, heat capture, and more. [audio mp3="https://orionx.net/wp-content/uploads/2024/09/090@HPCpodcast_IV_Lenovo_Patrick-Caporale_Memory-Centeric-Infrastructure_20240924.mp3"][/audio] The post @HPCpodcast-90: Lenovo on Memory-Centric Architectures – Industry View appeared first on OrionX.net.

There was a theory that as data sizes increase, the predictability of access, both temporal and spatial, would improve. And if so, then caching would work just fine. How has that changed with the new applications? You really now got to look at it at that kind of larger scale, where you now have processing power and compute that is not just within the central CPUs, but again, spread out across the architecture and really optimize the workload. And you mentioned how things are changing so rapidly with these new technologies coming online, presenting so many choices. Generally, when you're at customer sites, are customers struggling with these issues?

From OrionX in association with InsideHPC, this is the At HPC podcast. Join Shaheen Khan and Doug Black as they discuss supercomputing technologies and the applications, markets, and policies that shape them. Thank you for being with us. Hi, everyone. I'm Doug Black with the At HPC podcast. With me is my podcast partner Shaheen Khan of OrionX.net. And our special guest today is Patrick Caporale. He is Lenovo Distinguished Engineer and Chief IO Architect in the Infrastructure Solutions Group. And Patrick has been at Lenovo for 10 years. For 20 years before that, he was with IBM leading into the acquisition. Patrick, welcome. Great to be with you today. Let's talk

about where we are today with advanced HPC AI systems and your area of focus. Some of the new architectural developments and trends that you're seeing as we work to handle more and more massive data sets required for state-of-the-art workloads. Please, what are some of the most interesting trends that you're seeing these days? Yeah, first, thanks. Great to be here with you both today. It looks discussion, and it's really an important discussion, an exciting time to be leading IO architecture. So much is coming into the industry as new and of different opportunities for workload advancements and application performance. In the I-O arena, I look back and I look at the last decade, about 10 years ago, there was a period of time where the interconnects

were a bit more stable. We had things like PCI Gen 3, AR4 for a good part of that decade. But in the last five years, there's been a really flurry of activity around IO interconnects. And it's all, I think, driven by the fundamental background that data is exploding, right? We need to store it, we need to move that data close to the elements that need to process it. And to do that, we need to have interconnects that are really moving things faster, quicker, and actually changing the way we really process the data. I think that's really just an exciting new arena that we're getting into in the industry. So I'm super excited about this conversation. I always remember that definition of supercomputing, that a supercomputer is what

makes a CPU-bound application IO-bound. And if you talk to the practitioners, it's always IO. That's really where all the complexity is. So I'm really delighted to have a chance to double click on this a few times. But one way I see this is really starts with memory and memory buses goes into memory hierarchy, and then the memory interconnect, and then the networking, and then they all lead to storage. And as you said, there's just a lot of activity across that entire spectrum, right? So maybe we can just start from the top and really just talk about the memory part and what's going on in that world. I think that's a great place to start as well.

We've seen advancements in the processing capability, whether that be things like core counts or the speed of those cores, And you have to feed it the data. And I think the first tier in that kind of hierarchy, as you said, is the memory. And historically, memory has been something that you tether directly to, say, a CPU type device where you have interconnects that are relatively fast and close in proximity. And that has provided a very good solution for many years. I think as we expand and we see the growth of one, the data, but also different types of compute elements, not just CPUs, but now GPUs and other processing units, ensuring that memory is available and

processed in a way that provides the application performance, especially in AI solution sets. That is where that first tier is really seen in those last five years of changes with new interconnects, CXL, ExpressLink, and other things, even in just enhancements on the DDR interconnect itself, moving now to DDR5 and new generations of direct enablement right on that DDR bus to enhance and improve things like bandwidth, latency, and overall performance of your application. Now, computer architecture struck upon cache and caching as a way of hiding the latency between different layers of, let's say, storage or memory. Why is it that technology isn't able to keep up, that you actually really need to go to more, I wouldn't call it exotic, but more complicated technologies?

In a lot of ways, those really high-speed caches, they're just limited perhaps by the capacity that they can handle. Again, as we've mentioned, the massive datasets that we're starting to see, you want to be able to handle larger datasets, so you want to be able to expand out interior memory in a way that those local caches will still provide the most optimal and highest access rates, whether it's bandwidth or latency. But again, it may be just operating on a very smaller capacity where you start to see the interconnects providing the ability to have larger data sets and in-memory operations is, I think, critical and why the cachesaches worked so well for, and it's still necessary in system architectures these days, but these advancements elsewhere is just to handle

the larger data sets. So there was a theory that as data sizes increase, the predictability of access, both temporal and spatial, would improve. And if so, then caching would work just fine. But how has that changed with the new applications that causes such demand for really high speed, right there, storage of this size rather than smaller size? Oh, that's a great question. I think systems architectures are evolving. We're seeing, as I said, the processing units

become a bit more widespread in terms of it's not just centralized anymore. We're definitely seeing, we used to look at it in the ways of offloading, right? And we might offload certain functions, maybe on the network, we have network offload. Now we're starting to see entire processes where again, it's a GPU, it's a DPU, it's really providing enhanced compute capability and having that access of a kind of centralized memory or talking more about a memory centric type of approach of architecture where you start building out the ability for all of those different processing units to have access in a cache coherent manner. Any system that had two or more CPU sockets generally had cache coherency there. And architecture such as NUMA, the non-uniform memory architecture, provided that capability of there might be different latencies crossed from one CPU boundary to the other. But you really now got to look at it at that kind of larger scale where you now have processing power and compute that is not just within the central CPUs, but again, spread out across the architecture and really optimize the workload that the customers are after.

You mentioned memory-centric computing, memory-centric approach. How do you define that in just broad brush terms? Yeah, at a high level, I think that's one of those motivating factors that I think the industry has been driving to find solutions for. And it's really, first and foremost, I think, starting to look at processing data in place. Today, a lot of data is moved around. It's moved from one place to the other. It's brought into memory. It's pushed out of memory. Being able to drive to where memory can be retained, whether it is in cache or it is in memory, but then processing it in place, that's really the focus. It starts with an underlying at the protocol level, right? So some

of these new interconnects are changing the semantics of the protocol to take advantage of that capability. But it really goes to processing the data in place, changing from read and write operations, load store operations, having cache coherency so that regardless of how much access there may be to a certain memory region, I can continue to use that memory region until it's perhaps invalidated by some other access. I think that's really the memory-centric approach that I see. Patrick, some of these trends we're talking about, one thing, pretty common observation, but the media, the vendors, the media and industry analysts are all ahead of actual users and customers and what they're doing.

And you mentioned how things are changing so rapidly with these new technologies coming online, presenting so many choices. Generally, when you're at customer sites, are customers struggling with these issues? Would you say that you're out ahead of them, pulling them along? Or what is the dynamic as far as where the customer actually is? Yeah, something that's exciting to me could be very daunting to system administrators. You're absolutely right. And it starts from that. First, have a conversation like, yes, this is coming. There's going to be more opportunities to look at system architectures and that more traditional way of doing things. We now have different knobs we can turn and we can

look at doing things differently. And having that upfront of here's the technology that's coming, things are usually in, especially in HPC, I think a lot of that leads the forefront of some of the technology innovations because you're really trying to stress the maximum amount of computing power, solving some of the greatest challenges. And that is critical to the technology deployments that we look at from Lenovo. But really having that conversation is, yeah, breaking down all those very technical things that we're talking about here, but just saying, look, this change is coming. Some will be here in the next year. Some might be three years out. Some might be further. All of this is a journey that we're on together, right? I think we would all agree that things

like the data sets are increasing. We're seeing that processing become much more disaggregate, not just localized at a CPU, but we've got to take advantage of this. And then once we get that basis underneath us, which path we might go down or which interconnect this one or that one, there's still choices. And I think from a Lenovo perspective, as I look to build out those system architectures, making sure that we have good fits for the end user is really critical and really starts with that conversation. Yeah, I was going to say that a big part of Lenovo's job, of course, is to help customers absorb these technologies.

I would assume that's true. We certainly have heard from the chip vendors. They've really accelerated the pace, the cadence, which certainly new chips are coming out, but it looks like it extends across a lot of these technology fronts. Across that IO space, I really focus on it. It starts with just going back to those interconnects that are provided by the CPU, silicon, and other devices, but everything else is looking to enhance at the same time. We've got these approaches in memory. As I said, the DDR interface is rapidly improving in terms of its capabilities. We've got new technology coming such as multiplex rank DIMMs for doubling the bandwidth on there. And then even storage,

right? I mean, rapid improvement on networking doubling. And we're not doubling from going from one gigabit to 10 gigabit. That was 10x, but now we're going much faster, 200 to 400 to 800. And then beyond that, very near term, everything is taking, unfortunately, more power. So our systems, we are absolutely looking to be efficient in how we capture all that heat and how we provide solutions to provide that. Performance generally equals power, and power means we've got to provide cooling. And all that together is kind of a packaging challenge that we look at and really bringing all that together to make an offering for a customer patrick i want to go back to cash coherence and when the

beowulfs and mpps and clustering started in hpc a few decades ago, at least in part, if not in whole. It was about, I don't need cache coherency. I can just parcel it out to all these processors and I'll get the results back and reassemble them. And I have more apps that lend themselves to that than not. And I can harness the whole kind of a 10,000 chickens versus 10 horses analogy. But it looks like it's back and it's back with vengeance. And now suddenly, are we seeing a renaissance, a revival of S&P and NUMA, as you mentioned, but fundamentally cash coherence, either in a centralized or decentralized way? And if so, what is driving that? Why is this situation different from what we had before clusters showed up? I think you look at the two approaches of scale up versus scale out.

And I think there's a renewed interest in that scale up capability in architectures these days, and not just the CPUs, but again, for example, large aggregated together GPU clusters, right, in a scale-up manner. And I think in a scale-up domain, that cache coherency is really, because by doing the scale-up, that entire cluster is wanting to be doing the same work effort, right, and working on that same data set. And being able to have that cache coherency, sort of any one of those parts of clusters does make a change, notify the others and validate the cache. I think the scale up part is really strongly coming back, especially

with the AI type. So we've seen with large training sets that are looking just for massive amounts of scale up. I think our Lenovo solutions offering the ability to get massive eight-way AI GPU-based training platforms. That is, I think, driving a coherency model that is different than a scale-up approach, which could be done through a variety of interconnects, right? You've got your more traditional Ethernet or InfiniBand type of approaches and some new on the horizon. But I think that's where the coherency, I think, domain is really around the scale-up and all parts of the cluster now really wanting to work on that data set together.

I like the return of SMP, if you will, after having been written off a couple of decades ago as too expensive and too complicated and not necessary. And suddenly it's like very necessary. So that's nice to see, right? The other question really was about the gap that was always growing between CPU and memory speeds, and then between memory and disk speeds. And then that led to the whole filling up all of that with various caches and other forms of intermediate storage. Do you see those gaps continue to grow? Or do you think the gaps are getting closer and as a result some new

architectures can come about? Yeah, I think in some ways where there are gaps, the industry is looking to actually fill those. As we've talked to CXL, Compute Express Link Interconnect is one way to provide a memory tiering approach where you have today really high bandwidth, low latency, accessible memory, whether that's integrated into the compute element or let's say a DDR interface, right? And talking something in the order of sub 100 nanoseconds latency. Today, new architectures, if you go to another SMP or another socket, you incur a penalty there, which is a latency penalty. And that's well known by the kernels and applications that take advantage of new architectures. But to expand that to larger datasets, larger amounts of memory, you have

to fill that gap because today you might go to storage, right? And you might have NVMe storage, quite fast storage, but you're still on the order of maybe 340, 400 nanoseconds there. So you've got that gap. You've got local memory, and then you've got your storage. Something like CXL can provide memory technology, memory semantics in a way that tries to target almost that, quote, one hop away. So it's as if you were to access memory perhaps across another CPU socket, but instead you're actually going over a CXL link to access memory. It's actually filling that latency gap between the local memory and, say, ambient storage.

So I think that's a way that applications and workloads can take advantage of existing NUMA architecture and fill the gaps in latency by having new approaches like CXL. I want to ask you another question about technology before asking you about Lenovo's contributions to leading edge state-of-the-art. You mentioned the semantic impact that some of these memory technologies have had on how you take advantage of them, how you even code. Something that is in that vein is persistence. And if you're using storage class memory, which applications do have a general expectation of some performance attributes of memory, even if they are of different latencies and tiered approach. I think the main thing is, as the data sets increase, the more we can access

and the more application, as the number of cores and the processing power of those cores continue to go up, the more they have high-speed access memory. And whether that memory is near or far, it can provide advantages over pulling from storage. Storage is necessary, right? Because you're not always operating the entire massive dataset all at the same time. So it's that tiered approach where, you know,

applications wanting to access a certain large dataset set that's still a part of a much larger data set. I think we see the ability for applications to take advantage of that. I think HPC is a good example of scheduling certain jobs and activity that get run and dispatched and having access to that amount of memory for that amount of time and then dispatching that job, running the job, completing that job, and then handing back any available memory for the next data set to be loaded in, and then the processing of that workload in memory at that point in time, I think comes back to that approach of that whole memory-centric model. And I think we'll see more of that in years to come. Patrick, I think we were chatting earlier. I recall a presentation at ISC, I think it was 2017, around this whole theme of memory-centric.

And the notion is that we're approaching a day where there will be a decade of corporate data in live memory, which is pretty amazing. But by the same token, there's discussion that we're running up against the limits of some of the fundamental technologies of systems required to run these workloads. Are there areas in technology development, maybe in the interconnect area, for example, where you think a breakthrough could be approaching within two, three or four years? One, it's an interesting observation that you mentioned. Yes, there may be a time that it does come true. And I think all of this new technology that's coming in, I think we're all bounded by things

that are very basic, which is how much power is needed to generate all this compute power and how do we effectively cool that power because it will generate heat. And the second part was, there's also talk that we've hit the limit on some of the key enabling fundamental technologies that will allow these advanced workloads to continue to advance. I'm wondering, are there areas, say in the interconnect area or wherever on the technical landscape that you're hoping for or looking at maybe an imminent breakthrough within three to five years, 10 years that will really push us forward. And with that in mind, Patrick, I'm thinking maybe optical IO integrated in chiplets, or it could be cooling, or I hear about plastics,

plastic interconnects, replacing copper, whatever it might be that you're excited about. Now, on your question on what's hitting that wall, sure, that's certainly all this new technology. We've got to continue to look at, well, what comes next and what is the area that a breakthrough might need to happen? Today, we're still looking at a lot of these interconnects passing across electrical interconnects and connectors. And I think there's a new wave of things like optical IO and chiplet architecture

and different way of breaking out that IO architecture in a way that kind of may take us to the next thing. That's going to be a key area the industry is certainly going to see. Certainly there's optical interconnects today, but actually bringing that all the way back and down and how far down into the platform can you get those optical interconnects. I think that's another way to just continue to extend the life of some of these interconnects as they move forward. As you want to go faster, you want to have lower latency. The movement from electrical to optical type interfaces are certainly going to be ones the industry will look to address in years to come. Maybe this is a good time to focus on Lenovo itself and tell us if you would,

what you're doing that excites you that is pushing the envelope, driving new technology. Yeah. As we look at all this new technology, it's important to know that offering out the solutions to the customers, we need to package that. We need to provide the ability to give choice in these offerings. And that's really key. A lot of this new technology, as we talked, can be daunting. But Lenovo, our platforms look to provide a wide ecosystem of devices and capabilities. And all of these elements that we've talked about today, they all want to have a little more power and they need to have appropriate cooling. And I think that's where a main focus, especially in the HPC arena, as going back to 2012, first liquid cooled deployment at

LRZ, that really was perhaps considered risky at the time where, you know, moving really into a liquid cooled approach to do cooling, not just blowing air across hot devices. Liquid, by far, is a better conductor of heat than air is. And our Neptune warm water cooling technology, now in our sixth generation, is really one of the ways that we look to bring all this technology into the platforms and make that available to end users. Yeah, I've written about your Neptune product line and been amazed that literally water that's 120 degrees is used to cool systems. In my mind, that would heat the system up,

but good on you that you figured that out. Yeah, warm water cooling. Yeah, it's obviously an area we have deep history on and one that we'll continue to see advancements in. We started more humble beginnings with just looking at the CPU and memory, but we've now expanded that out to capture heat from the networking, the storage elements, even the power supplies, targeting up to 98% heat capture. It's not just about the heat capture. It goes back to, we talk about liquid is a better conductor of heat than air, but a CPU that runs 200 watts, 300 watts, or maybe 400 watts and more, to cool those, you might have hundreds of watts of fan power in a system.

And that's usually not something that's really talked about a lot because you're really more focused on the specs of the CPU, and they're usually very clearly listed. But when you start looking at how much fan power just to move air, you can actually recoup that in an actual energy saving. Our solutions look to recoup up to 40% of that energy cost by replacing spinning fans with liquid cooling. And that's where we have that deep history in HPC solutions going back more than a decade now and moving not just from CPU and memory, but capturing all the heat from the platform and really saving energy costs. Excellent. You're definitely right up there in that technology. Now, the impact of this whole IO infrastructure on cooling and heating, that seems to be something that is either

not quite a problem. It either wasn't a big problem, or it was neglected because there were other fish to fry. How do you see that becoming more of a requirement for data centers and customers to keep on their radar and actually address? Yeah, clearly, a lot of the attention goes to the performance, right? And you can easily measure performance as it moves generation to generation. But with performance comes more power generally. And with more power, in some cases, you need to keep things cooler to run faster and more performant, right? And those two things are moving, therefore, in opposite directions. So to go faster, you need to keep things cooler. That's why that technology that we bring is, hey, we're trying to take care

of that for the customer providing this technology. And a lot of the things you read on the spec sheets are going to be all around the performance. And we need to enable that performance. I think that's the key trend is it's just the massive performance power that's out there. And not just in terms of CPUs, but as we talk about all this memory and all this memory technology and storage and networking, everything is absolutely looking for more watts and in more watts, you need to be able to cool it and keep it cooler. Excellent. This also leads me to think about non-Von Neumann architectures. We talked about processor and memory, moving CPUs to data rather than data to CPUs. What do you see coming down the pike there

that is interesting to you? And is there anything that you offer today that can point in that direction? Yeah, we've seen trends there even in storage, right? Computational storage, for example, moving the compute to the storage, right? Is there a day that the compute moves to the memory? That may be something that comes to be in some system architectures as well. I think today, I think even just the advancements with Compute Express Link and CXL, I think that's the more near-term approach is to provide a way to have that cache coherency and some of that ability. And I think we'll be seeing that in the next few years, actually see more deployments. I think those will have to get, you know, the rigor of testing and going through

some deployments and understanding all of the metrics that are seen there and then making improvements on there. I think that memory centric model starts there and then ultimately may test those other areas as we've seen, like I said, storage and others look to go address and tackle. But I think memory, which is such an integral part of application in applications that we want to make sure these changes are agnostic to them so we don't have to rewrite applications from the ground up. I think that's going to be the key near-term important attribute here. Got it. Very good. Let's go way to the other end of the spectrum with tape and spinning rust.

What do you see there? Is that all the solved problem now and it's not where the action is? Or should we expect to see activity there as well? It may not be getting as much of the attention or technology leaps that we're seeing here in these other spaces. But again, back to that tiered approach for where your data is, you can go all the way down to, say, cold storage and still see very large, high capacity spinning disks are still available and probably will be for quite some time. Because again, there is a category of where you just want to store some data. Now, of course, as you go from there, you're going to have more higher speed access, warm storage and hot storage, and then memory and memory tier.

So I think there's a place for all of this. I think the advancement in some of this technology is more at the upper end of that tier at the moment and why you're seeing more, especially the memory in more recent days here. But certainly, I still see the variety of options all the way up and down the IO interconnect landscape here. And it certainly includes all that, again, down the IO interconnect landscape here. And it certainly includes all of that, again, based on the customer's need. Yeah. One other question, and this is about AI. Are the access patterns and data formats and just how that whole thing flows, are they settling? Or is that still such a changing world that you have to basically optimize these one at a time or close to it?

In the AI arena, we've seen actually some new advancements in interconnects, specifically targeting that workload. We talked previously about scale up versus scale out architecture, and we've seen Ultra Ethernet Consortium for scale out Ethernet to support large AI clusters as one example. Kind of that very focused approach in the AI arena as just more modern AI solutions are coming and having an industry offering that the ecosystem is coming together to support is one example. And to go along with that, more recently, there's been announcements around the ultra-accelerator one. You'll see industry companies getting together to provide scale-up acceleration of the interconnect for GPUs and other accelerators.

I think these are two examples of where we're seeing enhancements of the technology moving forward, very dedicated and very specific as to that AI arena, where it's not just for general- purpose workloads, but AI in particular. Patrick, we've seen, and we've mentioned this earlier, the cadence of new products from the chip companies, NVIDIA, Intel, and AMD, really accelerating. Does that pose challenges for Lenovo to incorporate those into your server line? That's one that we're absolutely looking to address head on because definitely having an

agile solution set that's providing choice to the customers out that want to use whichever their choice may be for HPC, AI, and other workloads. I think that's one we're absolutely looking to enable and provide support for. It can be challenging in terms of the cadence, but I think the market is demanding this at this moment, right? We're seeing the data sets, the need for AI training, and then inferencing beyond that is not going to, I think, diminish. And I think we're going to continue to see this type of rapid influx of technology. So from our system architectures and we build out platforms, we look to, again, rapidly enable that and come to market with solutions and maintain our technology system, Neptune, warm water cooling, and provide those offerings. Working closely with our partners who are developing the silicon and defining those chipsets that we can integrate and

provide offerings to. Maybe consider it a challenge. I started out, I think, at the beginning of our podcast here talking about it's actually exciting. So it's a really interesting time in the industry to see all this rapid movement. Patrick, of course, we're very much focused HPC and AI, but we've also seen much broader adoption of HPC combined with AI into broader markets. Talk a little bit about that, how you see this HPC Class A moving into the enterprise and the implications of that in terms of the systems that you're delivering. Yeah, great observation, right?

HPC is generally at the forefront of a lot of of technology pushing the envelope on performance and cooling technologies. But as we've talked, a lot of this next generation technology is, of course, coming, and it will come in environments that extend beyond just HV and AI and into the enterprise. A good example I can give is with our Neptune cooling solution. It certainly began with our HPC platforms and providing that. But we're now seeing that CPUs and accelerators are wanting so much power in more traditional 1U and 2U rack servers that we're bringing liquid assist cooling or even Neptune direct DRAM technology in those systems and more traditional think system platforms that we offer. That's actually, I think, on the horizon. So HPC, definitely at the forefront of all this technology. But as we build out our

solutions, we absolutely see a lot of this capability coming back. Now, maybe some of the most highest end CPU or the highest end accelerators are targeted for certain platforms, right? Accelerators today, well, we have, maybe 700 watts moving to one kilowatt and beyond, right? They're in certain specific platforms, but packaging them into more traditional 2U type systems, you still need some of the technology and Neptune provide that. So yes, I think that's an absolute area. We see the involvement of all this technology coming into enterprise and other markets. Excellent. Patrick, will we see you at SC24? What do you have planned for that event?

Yes, I'll be at SC24 Lenovo annually. We will showcase a lot of this technology, especially our Neptune warm water liquid cooling solutions. I'll be there and certainly look forward to meeting and seeing other industry activity at SC24 coming in November. Okay. We'll look forward to that. Wonderful. Thanks so much for taking some time to talk with us. Very interesting stuff. Yeah. Thank you, Patrick. It's been a long time since I wanted to double click on all this IO memory stuff. And it's been great to have the opportunity to do that with you who are so close to it and pushing a lot of the technology leading edge. What a treat. Thank you so much.

Thank you for having me. It was a pleasure to talk to you about today. Thanks so much. That's it for this episode of the At HPC podcast. Every episode is featured on insidehpc.com and posted on orionx.net. Use the comment section or tweet us with any questions or to propose topics of discussion. If you like the show, rate and review it on Apple Podcasts or wherever you listen. The At HPC Podcast is a production of OrionX in association with Inside HPC. Thank you for listening.