Utilizing Tech - Season 7: AI Data Infrastructure Presented by Solidigm - 4x10: Pathfinding Cloud Architecture for CXL with Dan Ernst of Microsoft Azure

Episode Date: January 9, 2023

Cloud architects have long wished for more flexibility in how memory is provisioned in servers, and CXL is finally delivering on a decade of promises. In this episode of Utilizing Tech, Stephen Foskett and Craig Rodgers talk to Dan Ernst of Microsoft, who is deeply involved in bringing CXL-attached memory to fruition within Azure. Dan was previously involved in the Gen-Z effort, and feels that CXL picks up the baton and brings valuable real-world benefits to cloud server architecture. Memory has a bigger impact on overall IT platforms than many are aware, and is already the costliest component in large servers. Per Amdahl's Law, it makes sense to look for cost savings in memory. And this is doubly the case because DIMMs only come in certain sizes so memory is usually over-specified. CXL memory modules cost a bit more than a DIMM, but this is offset by efficient right-sizing, and CXL is already cheaper than 3D stacked DIMMs. Hosts:   Stephen Foskett: https://www.twitter.com/SFoskett Craig Rodgers: https://www.twitter.com/CraigRodgersms Guest Host:   Dan Ernst, Principal Architect working on Future Azure Cloud Systems, Microsoft Azure https://www.linkedin.com/in/danernst/ Follow Gestalt IT and Utilizing Tech Website: https://www.UtilizingTech.com/ Website: https://www.GestaltIT.com/ Twitter: https://www.twitter.com/GestaltIT LinkedIn: https://www.linkedin.com/company/1789 Tags: #CXL #DIMMs #Azure #UtilizingCXL @Microsoft @Azure

Welcome to Utilizing Tech, the podcast about emerging technology from Gestalt IT. This season of Utilizing Tech focuses on CXL, a new technology that promises to revolutionize enterprise computing. I'm your host, Stephen Foskett, organizer of Tech Field Day and publisher of Gestalt IT. Joining me today as my co-host is Craig Rogers. Hi, Stephen. I'm Craig Rogers. You can find me on Twitter at CraigRogersMS. So Craig and I, for every episode of Utilizing Tech, from the AIs to the CXL episodes now, we've always focused on practical applications of technology. And that's a little hard to do when you're talking

about things like machine learning and CXL, because a lot of this stuff is still in the future. It's hard to put yourself in the present and talk about real practical, real world use cases and so on, if you're mainly talking to vendors of the technology or people who are trying to push it in the future. You know, Craig, it's been a challenge for us sometimes to kind of pull that information out, though I do have to say that our guests have done a nice job of keeping it real. They have.

As with every new technology, defined process doesn't exist. And large companies that need to abide to processes are having to figure these processes out now and dip their toes into the CXL pool. And it can be hard too, because as we talked about, I mean, the first platform that supports CXL was just released in November, the AMD platform. Now we expect that there's going to be another platform released real soon now, but of course, we don't have any definitive word on that as of today, January 9th.

So we'll see where that goes. But we do have a guest today that maybe can shed some light on the sort of thought process that goes through the minds of an architect that's looking at implementing this technology in the real world. So joining us today for utilizing CXL is Dan Ernst from Microsoft Azure. Welcome Dan. Thanks Steven, happy to be here. So Dan, tell us a little bit about your role there at Azure and what you're doing relative to CXL.

Sure, at Azure I'm part of a team that is doing pathfinding for our future architectures for the Azure cloud, future hardware architectures. And as part of that, I lead a team that's focused really on memory technologies and architectures. So everything from different media to different new protocols like CXL, and then trying to figure out how to best apply them and where we might be able to use them in our fleet. So let's start at the beginning.

As a cloud architect, as somebody who's really interested in memory technologies and memory architectures, when you first heard of CXL, what was the first thing that kind of popped into your head to say, wow, this technology could be really useful, how? Well, what's interesting is some of the conversations about these use cases, these technologies, started even before CXL came about, right? There have been a number of initiatives over the last, let's call it six, seven years

to try and expand the definition of what the memory bus looks like on computers, right? For the longest time, decades now, everything from memory controller in was on the CPU. And so the memory controller has been a domain that system architects can't mess with, right? You get what you get from a memory perspective. And so there's been a lot of pushback in recent years in particular, to try and get more flexibility on that. So you can think about things like prior initiatives, like the Gen Z consortium was trying to do a

similar kind of thing. And really what CXL was able to do was to get the real, it was to build the right coalition of companies to really bring it to fruition and to reality, right? And starting at a place that was really trying to address this ability to move, you know, the memory control point off the CPU and into the hands of system architects and the rest of the ecosystem to give us more flexibility. So it's pretty exciting technology from that regard. It's interesting that you mentioned Gen Z. I also know from your LinkedIn that you are also involved in Gen Z and the the I think you hit the nail on the head there that CXL has got the right

coalition of companies working together and that may give it a bit more traction than what Gen Z ended up with. But Gen Z solved a lot of problems for companies that needed memory pooling at scale. What's your thought on CXL compared to, say, Gen Z? Do you think they're doing the same thing a different way? What are your thoughts on that? Yeah, I think Gen Z's role was to really highlight what could be done. It was really to bring out some of these use cases and show them, show one, that they could be done.

And two, to show that some of, you know, that sets of them have some value if we were able to do some decoupling. From that point, there's a long technical conversation about what pieces of that do we want to start from in actual implementation for a real at-scale production technology, right? And CXL really picked up that banner and said, okay, let's start with these first few things and we'll kind of build from there. But it's built very quickly, right? And in fact, there was the merger where the Gen Z consortium has now folded into the CXL consortium.

The IP is now within the CXL consortium and CXL now can pull upon that IP as we're, you know, progressing the spec forward to try and adopt some of those other use cases, some of those other technologies that were developed within that framework and apply it into the CXL protocol if we find good use for it. So in the end, it's all technology that is being brought forward, you know, into CXL. So we should see some of the benefits around the approach of an iterative process where we started with one thing, we make changes, do the next thing. CXL could be thrown in some ways. Yep. And you've seen that already with, yeah, with spec version one, 2, and 3 of CXL, right? It started quite simple. And now we're talking about fabrics, right? Which are not that simple, right? But it has been this iterative process of like, let's look at some, okay, let's see what other new use cases we can solve. Let's find the protocol, put the hooks in to do that, and then we keep moving forward.

I'm sure future Dan would be able to talk about all the CXL version 3 things going on, but right now, what are you looking at as an architect within the Azure hardware platform, if you will? Obviously, it's memory pooling related, but I imagine the impact for Microsoft on Azure with CXL could potentially be very large. Well, in a general sense, I mean, the impact statement, I think, is a useful one to start with, which is that memory technology, memory itself is a really big part of IT. It's a much bigger part that it's often given a lot of attention for, right?

I mean, we've reported in some of our publications that, you know, we anticipate in the next few years that memory cost could be half of our, you know, server cogs, roughly, right? Just by virtue of the amount of memory we need, the fact that memory scaling is, you know, not really advancing, especially in the terms of, you know, cost is not really going down. And yet CPUs continue to advance and we get more cores. And so as a percentage of the whole, we're having to really invest a lot in getting the memory system up to meet the capabilities of the rest of the server. So because of that, any of us who are familiar with Amdahl's law know that if you want to try and optimize a system, you want to attack the portion that's the largest piece of that.

And so memory is definitely a case where we could have a pretty big impact by trying to address how do we get either memory that's, you know, in a memory architecture that gives us something that's more affordable or memory architecture that is more efficient, right? That's really sort of the memory pooling case. So these are the kinds of things we're looking at. Right. How can we apply just this straightforward memory attached technology in CXL and apply it in a way that really can help us do something we couldn't do otherwise? That's the interesting aspect. And as you said, I think a lot of people may be surprised to hear that memory is, I think it's safe to say, memory is the single largest component of storage. Even if it's not half the cost of the system, it's the largest component cost. Now, obviously, a lot of other things cost a lot of money too, you know, CPUs and storage and networking.

But memory is certainly right there. And anyone who's built a large system is aware of this because it gets very costly very quickly. And then the other thing that you mentioned, I think that's really key here when it comes to CXL is that inflexibility. We've talked about this quite a lot before. In order to keep those cores, all those multi-core processors, in order to keep those cores fed, you need a lot of memory. You need a lot of low latency, high performance memory, which means that systems now are having more and more memory channels, which means you have to fill all those memory channels. And since DIMMs only come in certain fixed sizes, there becomes a point where a system architect is going to ask himself or herself the

hard question and say, do I want to put, you know, what size DIMMs am I going to specify here? Because obviously you can't put too little memory in. So you always have to put too much, but putting too much ends up being a really big stair step. Like, you know, it suddenly goes from this to huge, right? I'll tell you a story from past life, actually. Before I was at Microsoft, I spent 10 years at Cray and my merger at HPE for a couple of years, and I was working on high-performance computing systems. And those systems often have very high memory performance requirements,

but capacity requirements are not huge. And as we saw high memory performance requirements, but capacity requirements are not huge. And as we saw the memory channels expand, we saw exactly what you're talking about, which is, you know, our VP of sales would call me repeatedly, like, is there a way we can find smaller DIMMs, right? Just find me a way to get less memory because we were over to get to the right bandwidth to get to the right, uh, RAS capabilities. We were overbuying memory by at least a factor of two, right? Um, now we can be a little more efficient about that in, in, in a cloud world because the, the constraints are a little different, but, but the, the flexibility is a big thing, right? Um, you know, the, you really only operate in memory DIMMs in factors of two, and mixing is not really good from a performance standpoint.

So having CXL as a way to decouple that, I think, is going to be a big step in the direction we need to take. And that's one of the use cases we're looking at. Yeah, absolutely. And even if you get away, I mean, we're hearing, well, what they're calling, I don't love this term, but they're calling non-binary DIMMs, which essentially just means kind of a half size, a half step DIMM, which is a great idea. I love the idea. But even there, it's sort of a total, I mean, it's, I don't know what the right word is. You can't underbuy, so you have to overbuy, right? And if you, and if, and, and, and so you only have so many choices and it's nice to see that CXL can maybe try to fill in that gap somewhat. Are you also seeing that those CXL

memory cards, are they more or less expensive than DIMMs? I mean, what is that going to look like? So just in a generic sense, the way to think about it is a vast majority of the cost of a DIMM is the media itself, right? The other parts in the DIMM do cost money, but relative to the memory itself, it's effectively zero, right? It's very little. If you can apply the same thing to CXL device, right? In that the media cost is going to transfer, right? If you're taking DDR5 on a DIMM, and now you're saying we're going to put DDR5 on CXL, the media cost is going to transfer, right? The controller costs. So there's an additional controller with CXL, right? That's going to do the protocol conversion

between the CXL protocol and the DRAM protocol, for example. So that chip component is going to have some cost to it, right? And and so I would say there's if you're comparing an an rdim to a um a cxl call it a cxl dim right a cxl module um the cxl module with the same media the same capacity will be slightly more expensive and that's sort of how we've been modeling it right just by virtue of there being more components there, right? More bigger components there. It's a small amount, but it is more. Now, the interesting thing is,

okay, if I'm comparing to Ardium, it's slightly more expensive. However, if I need a server with, for example, a lot of memory, right? Large in-memory database servers, that kind of stuff, right? Those servers need so much memory that they have to use, for example, 3DS DIMMs, right? DIMMs that use stacked memory on them. They're very dense, and they are also extremely expensive. The technology takes a lot of extra space and effort to build those out. So you're talking about a multiplier of cost per bit.

If you compare 3DS DIMMs to CXL memory, the CXL memory is going to be significantly cheaper. And so if you could, for example, use CXL to get your server system to a capacity that you otherwise couldn't get to without 3DS DIMMs, suddenly you have a pretty big win, right? So these are the kinds of trade-offs we're trying to play with, right? Of how do I get to a certain capacity? And before, our choices were, which DIMM do you need? And that's the choice we had. Now it's how do we mix these technologies

in a way that is both performant and also cost optimal? And we're also working outside of the constraints then imposed on the number of DIMM slots you can put on a server system board. You know, there's obviously a limitation there in terms of real estate as to how many slots. Yeah, it is challenging. The infrastructure for this stuff is going to be interesting to see because you want simultaneously, you want this capability, you want it designed in. At the same time, you don't want to burden a standard platform that might not want to use it, right, all the time.

And so you have to have some tradeoff between modularity. The nice thing about CXL, one of the really slick things about the way it was designed was it borrowed all of the physicals from PCI Express. And so we can use existing expansion slots, for example, to take care of all of the sort of physical constraint and flexibility standpoint. We can treat it just like a flash drive in the instance of plugging in as many as you think you need. From that perspective, it was nice that they used something that was already there. Yeah. It was actually really exciting at the CXL Summit to see these solutions already in play, in work, granted engineering samples.

They might not have been ASIC. They might have been flip-chip PGA, but they were there. They were functioning. You couldn't even buy a server at that point that was CXL capable, but we had multiple companies showing their wares, if you will. It was good to see. Yeah, people are excited about it.

Like I mentioned before, system architects haven't had the ability to have this flexibility in a long time, right? Since the Northbridge went on to the CPU mini for those of a certain age, right? That's a long time ago. So it is interesting and people are excited about it. So tell me a little bit about, I don't know if you can specifically talk about what you're doing with Azure, but as somebody who's in this space, somebody who's really taking this technology and putting it to practice, what's the real deal? Is this really making an impact? And is this really allowing you to do something you wouldn't have been able to do before? Sure. I think there are definitely cases where CXL is going to have an impact. And I would balance that against,

there are places where people might blindly apply CXL that might not have the impact they would like, right? And so I think, I like to think of my job as not just thinking about what CXL could do, because CXL is a really very flexible spec. It's a really powerful specification. I think the key thing for me is to figure out what should be built. What cases are we actually going to get value out of it?

And what cases are interesting but don't really come through in that regard? So one example of this is some of the work we've published around memory pooling. We've been doing some projects investigating this with our compatriots in Microsoft research, looking at if we were going to use memory pooling in a public cloud, what's the sort of the problem statement for that? What is our goal for it? And then what are the things that have to happen from a system stack perspective for us to really leverage this into something beneficial, right? And I think this is a, you know, the key takeaway from a lot of that was if you're going to use something like memory pooling. So memory pooling, the idea is

you would have some memory attached to CXL, but that CXL memory would be attached to multiple servers. And so those servers could then fungibly take that memory and pull it into their servers, you know, not physically, right? Logically pull it into their servers for use when they had high load, for example, right? So you can sort of move memory between servers in a way that lets you perhaps deploy something a little less aggressive, right? So you talked about in DIMMs, you always have to overbuy, right? Well, maybe you underbuy your worst case, right? Where there's a peak workload that needs all this memory. Maybe you underbuy that use case, but the one that, you know, but you have enough memory for just about everything else.

And then you apply, you have a memory pool that you can use to serve that peak use case. So again, it helps you sort of have that flexibility against the harmonics, as we like to call them, the dim harmonics that you would normally be held to right um but you have to trade that up you really have to know what your workloads are doing right this is a really important piece of that if you don't know what is the memory footprint usage and

bandwidth needs of of my workloads right you know as a you as a, you know, if I'm a, an enterprise IT person trying to build up, you know, where am I going to apply this? You really have to know what's, what's running, what your utilization is, and if this is a match and then how does that match work relative to, to what the, the possible configurations are, right? So it's really challenging and it's fun from my perspective because it's a full stack problem, right? You have to know the applications, you got to know what middleware pieces are running to support the performance of the application.

And you have to know the hardware pieces of what's available and what's possible, right? To make that all come together to actually be beneficial. And that's long been the case of pooling for storage as well. That's my background. One of the things that people get from shared storage, which is the analogy of memory pooling, except on the storage side, is you basically get better features that can be shared more efficiently among multiple servers. So as opposed to having to buy,

I don't know, like, you know, X terabytes of high performance, high feature, high function storage in every server, you buy a much larger system and then you share it out to all of them and then they all can use it. And even if it's a little slower or it's a lot more flexible and it allows you to write size

and to sort of, you know, grow, do you think that that's what's going to happen with memory pooling as well, that there are going to be features that are going to be brought to bear that aren't just capacity? Yeah. And that's one of the benefits of, again, pulling the controller piece off of the CPU, right, is now you have some flexibility of what that piece of silicon might be able to do, right? There are options there that people have been exploring for a long time, but haven't had the ability to actually try and practice. I think the one, the one, the analogy to storage pooling is I think very, very good, right? That's a, that's a close analogy. The one thing place where I would be careful about that is that the performance requirements is different, right? And, and sort of the, what I would call the software layering of it is different, right? Storage pooling, you're dealing with a storage system.

You're talking to it through POSIX IO or whatever mechanism you're using. And you can hide an awful lot under that, right? You're hiding network transactions under that effectively, right? CXL is a very, very hardware tied at its memory related protocol. Right. And so anything that's happening on that remote piece of silicon is going to be probably hardware, not software. Right. Because if you're in software, you've blown the performance needs out of the water. Right. So, you know, there's a little bit of a difference there in terms of how you would add features. And I would add, because the performance requirements are a lot tighter for memory, the scope and scale of how much you're going to share or what distance you're going to share, right?

You know, you might be pooling storage amongst racks, right? You know, from our perspective, you're going to be pooling memory likely very locally, a few servers, right? You know, from our perspective, you're going to be pooling memory, likely very locally, a few servers, right? Not much more than a chassis, you know, a small sub-chassis size, just because you want to keep the latency low, right? You don't want all that overhead, right? But again, yeah, there's definitely, again, a lot of opportunities. CXL gives sort of control of those opportunities back to architects outside the CPU complex. And we've mentioned now a couple of things. I want to ask you if you have any, I don't know, experience with this. The performance profile, the performance characteristics of basically local CXL versus remote CXL.

And I know that some of this stuff is still in development. I know that maybe you can't talk about it much, but I think most people would assume that there's going to be a big performance hit to CXL memory. And we've recently heard from a number of different companies now, including now that AMD has launched their platform, that local CXL memory on the CXL bus within a server

doesn't perform any or noticeably different from, you know, remote NUMA memory. Is this anything that you can talk about? Can you contrast or give us any kind of sense of what's the real world? Yeah, we can talk a little bit about it. You know, in terms of real world measurements, we're still very early, right? As you mentioned, there are people with silicon, but it's engineering samples, everybody's still trying to tuning everything up, right? But the statement that Mahesh made from AMD about latency is pretty accurate, we found. That's always been the goal, right? We want this to look like, you know, far socket latency, right? In a two socket node, the latency of going to memory and the far socket

so far we're seeing that hold true that seems to be be pretty accurate and you know more so again from that point okay does that matter for what your applications are doing well it's going to depend heavily on your applications right there are certainly workloads where that's that's painful right and and will take your performance impact from that. But there are definitely workloads that are much less latency sensitive or not particularly memory intensive that will run just fine. It might be a couple percent degradation, but not something that is really going to overwhelm

what your benefits might be, right? So, so far in the performance angle, we've been pretty happy that it looks like things are going to land the way we thought. Now, remote CXO, which is if you were to, say, put a switch in and have CXO devices hanging off a switch, we see that as being, again, you're just going to tack latency on top of latency. Right. So if you can think of a far socket, latency is, you know, upper 100 nanoseconds, maybe just shy of 200. Right. You could think of every switch you put in the middle of that might add another 80 to 100. Right. Just ballpark in here. Right. You know, that's really sort of the limitations of switch of of, you know, edge crossings of silicon plus a little bit of switching, right? So, again, the usability of latencies that look like that, right, you know, 300 plus, you know, nanoseconds, people have investigated that, right?

That was actually not that far off from the quiet latency you might have seen with something like Intel Optane, right? But from a perspective of, you know, it goes back to if you know what your applications are doing, then you can make this tradeoff, right? You know, if I know my applications don't really care about that much latency, then I might be looking at that if I can find value in that architecture, right? We tend to think about our more general purpose fleet as being very broad based and we care about tail performance, right? Well, if we care about tail performance

and there are applications out there that are gonna hit, you know, 300 nanosecond memory, we're probably gonna be less enthusiastic about that as an example, right? But that's sort of a specific case, right? You have to make this trade-off call of how much do your applications care? How much benefit can you get from that architecture? And then, you know, try and find a way to resolve that based on the metrics that you care about. Can you shed a light on how, say, Windows servers or the custom hypervisor running the,

I know it used to be thousand node clusters in Azure. Can you talk about how those servers are addressing memory with other guests who have seen that it can be a shared system mode where it's virtually transparent or could be presented as a device. How is Windows going to see and interact with CXL attached memory? I guess the way I would put it is they're working on figuring that all out. Certainly the short version is there are going to be ways to just expose it as system memory, right? And if you want to use it as generic memory, you can do that. But, you know, if you look at what Linux has done,

for example, there's also ways to do, you know, designated as sort of special purpose memory or forget what their actual word is. You know, so you can have some a little more dedicated APIs to, you know, make sure you're an allocation to the right place. But the philosophy there is very similar. It's memory. We want to treat it differently than your regular local memory. Whether that's a MUMID delineation or a special purpose designation or whatever. But there are options there for sure that are investigated for for when we get this sort of into real production deployment.

Yeah, I didn't want to mention the L word there. All right. That's you know, but it seems obvious that that there have to be some type of support there, you know, from the from the CXL world, we want all this stuff to work, right? And I'll add, even on Azure, right? There's a lot of Linux that runs on Azure, right? For sure.

So we want that support to be broad-based. And we're really excited, by the way, in a lot of what's happening in the community around CXL, in the software space. You know, we're seeing people come up with all kinds of different ideas about how you can manage multiple levels of memory. Everybody's going to have different needs

around where that management should happen, should it be in a guest, should it be in a host, should it be in hardware. We want to have all those options out there because people's different use cases are going to have different needs, right? You know, we've seen presentations from VMware on their hypervisor work. We've seen presentations from Meta on what they're doing in user space with analysis and page placement, right? So that to me is also really super exciting, right?

To see the development of people learning how to take the software and take advantage of this well and make it as flexible as possible. on utilizing tech. We also had Memverge join us on utilizing tech and both of them talked a lot more about the software features there as well. And I think we can safely say that all of the major software vendors and software operating system projects are focusing on bringing this stuff to fruition. And I'm sure that Microsoft is as well. So Dan, I know that you're the memory guy, but looking forward, I always have to ask, is CXL relevant beyond memory? And

where do you see it going next? I think that's a great question. And the answer is the specification absolutely expands behind, you know, goes beyond memory, right? There are a lot of cache coherence mechanisms defined to out, okay, where are accelerators going to be? Where are they going to land in that space? I think it gets even more interesting now as you're looking at something, it basically, one of the things it exposes is, hey, you can use this physical interface to transport CXL. So if you wanted to put accelerators in your package in a design like that, it provides, you know, all the standards are there to make that happen quite simply. So that's a, you know, a step we think in the right direction and where in the longer term, I mean, dealing with cache coherence, dealing with memory on an external bus is not simple, but it's much simpler than dealing with cache coherence on an external bus.

And so I think the accelerator stuff is still catching up in that regard. They need to spend some time making sure they can get the use cases right and get the architectures right. But I think it is definitely a goal of CXL to serve those uses as well. Yeah, it's going to be really exciting to see where that all goes. And I know we're trying to keep it real here, but it is also kind of exciting to think, and for somebody like you who's doing this kind of cloud architecture, to think about how, if this all works, how it could change the shape of servers. You know, you already mentioned, you know, a chassis of servers that are having, you know, multiple servers sharing a pool of memory. You know, we talked about UCIE and how that can change the shape of the processors. I would not be at all surprised if a future Azure architecture looks incredibly different from the current architecture in terms of the shape of things and how things are specified and what tools you have as an architect to put these systems together. I agree with you, Stephen. I think that the direction we're heading is

making servers a lot more interesting, right? It's not, you know, the hyper-converged pizza box forever, right? I think we're starting to learn to treat the rack and the data center as a system, as opposed to just, you know, one server at a time. And when you do that, you start changing how you want wanna connect things. You start figuring out how to share things. Where are your data usage levels? What scope and scale? So yeah, I know it's a pretty exciting future

to look forward to. Computer architecture is certainly not boring in the system space these days. Yeah, absolutely. Well, I hope that our listeners found something here that they can look forward to and also found maybe some inspiration that this technology is good enough for Microsoft. Maybe it's good enough for you now that these things are real. And also, Dan, I would love to reconnect with you maybe later in the year or something when we are able to sort of see what this really did once it's all out there in production and use, you know, how it's really transforming things. So I look forward to future papers from Microsoft. Also, one thing I'll say, too, is I really, really sincerely appreciate Microsoft sharing their research on this and sharing these papers. We're going to link this to the paper that Dan mentioned here in the show notes.

It's so valuable that we can all benefit from their work. So kudos to you for that as well. Yeah, it's been good. It started a lot of really good conversations. And, you know, we're really interested in an ecosystem understanding the use cases here and starting to build things to serve them. So yeah, it's been good. Please check it out.

Excellent. And thank you so much for joining us. Dan, before we go, where can people connect with you? Do you have anything on the roadmap or anything you're proud of recently? Well, I think the big one is we've started publishing work and then people have taken notice. We've given some talks. I gave a talk at the OCP Memory Summit that I think a lot of people

saw as a, here is why memory pooling and what the basics are. So check you know, check that out. You know, coming up this spring, I think we'll be we'll be attending the, you know, a couple of conferences and talking more about this. So stay tuned. We definitely will be. Craig, anything to add here? No, it just it's really interesting to hear from a juggernaut such as Microsoft, you know, and obviously through Dan, but it's, you know, such a big player in the IT landscape, you know, they're gargantarian and they're here, Utilizing CXL. If you enjoyed this discussion, please do subscribe. We can be found in all of your favorite podcast applications, and we would love it if you gave us a rating or review. We've got quite a few subscribers, actually. We were pretty excited to see the listenership for this season. This podcast

is brought to you by gestaltit.com, your home for IT coverage from across the enterprise. But for show notes and more episodes, head over to utilizingtech.com or you can find us on Twitter at Utilizing Tech or on Mastodon. Just look for Utilizing Tech in there. Thanks for joining and we'll see you next week.