In The Arena by TechArena - Broadcom's Hasan Siraj on Why the Network Is AI's Computer

Episode Date: July 29, 2026

Networking has quietly become a decisive variable in AI infrastructure performance. In this episode of The AI Hedge, host Marc Austin talks with Hasan Siraj, VP of Products at Broadcom, about why the network functions as the computer for modern AI clusters.

Hello, I'm Mark Austin, founder and CEO of Hedgehog. This is the AI Hedge, where I talk with industry leaders about AI infrastructure and how to hedge risk with really big financial decisions around AI infrastructure. So with me today, I have Hassan Saraj, good friend of mine that I've been working with for several years now at Broadcom. Welcome to the show, Hassan. Thank you very much, Mark. It's great to be here. And thank you for having me. and thank you for the partnership over the last several years.

Yeah, it's really been great working with you and your team. It'd be great for you to just start out by telling everybody a little bit about what you do at Broadcom, maybe even explain what Brodcom does. I don't know if everybody listening understands Broadcom's really significant role in the industry, what you do at Broadcom and what you've been doing throughout your career. I think it'd be super interesting for everybody. Sure. At Broadcom, I lead the strategy and product management team in what we call the core switching group.

This is where we build routing, high performance NIC, open networking software portfolio. Specifically from a product perspective, some of your audience may know like the Tomahawk, Jericho, Trident, the Thor Ultra, kind of the line of products. And also we have a team that, a pretty large team that works on Sonic, SIE, the SDK and embedded software.

We work very closely with the hyperscalers, but also with the enterprise and service provider customers, also interact a lot with our ODM and OEM ecosystem, including the startup ecosystem, including the partnership with you. Yep. And over the last four to five years, we have been hyper-focused in helping customers build AI infrastructure based on open standard technologies like Ethernet.

But it's been 8 years at Broadcom. And before that, I've also led the routing and switching product management teams at Cisco. You've got a deep background in networking. And as we talk today, we're going to talk about the role of networking in AI infrastructure, and it's a pretty material role. Broadcom's piece of this is a silicon maker primarily, right? But then different manufacturers use to create data center switches that connect all of these GPUs together. Absolutely, right?

We build the silicon, which is AI infrastructure, four enterprises, for service provider networks, or basically the cloud infrastructure. and we don't build the routers or switches themselves, but we have an ecosystem of partners who will take these, build the hardware, run their own software, and then partners like yourselves who are also doing the orchestration

on top of all of this. Yep, great. And you talked about open software as well. I think maybe not everybody understands this, but like switches have operating systems on them as well. Can you kind of give everybody a little bit of an overview of what that means? Operating systems play a very important role on exposing all of the functionality that we have in our silicon.

We support a whole variety of operating systems. We will have OEM vendors who have their own operating systems that run. There are disaggregated OS partners that who have these. And then you can also run this open source, which is Sonic seems to be a very popular choice. We have ourselves been very involved in the Sonic and Cy projects ourselves. one of the largest contributors. Right. Great. We're in, I don't know if it's the middle or it still feels like early phases

of this massive AI infrastructure buildout, right? I was just listening to the news this morning. It was talking about SK Hynix being available in the United States markets. It was talking about meta, becoming an AI infrastructure provider, I think probably with your company's help. That tech companies now,

because of these sort of valuations on infrastructure providers, your company included, represent more than half of the S&P 500's value now. So it's clearly in the news, presumably also a strategic priority for nearly every industry. From your perspective, what's fundamentally changing about how organizations think about infrastructure because of AI? Because you've been in the infrastructure game a lot longer than this eye boom.

Yeah. So if you look at the infrastructure build out over the last, let's just talk about data center infrastructure. build over the last two or two and a half decades. The focused had been on virtualization. So you were basically had a bunch of compute and you were virtualizing this compute and running multiple applications on this. And that had been the focus. But with AI, when people are finding out is it's fundamentally a very different workload. These models, they don't fit in a few cores of a CPU, tens of thousands of cores of a XPU or a GPU. You need tens, hundreds. And with these models becoming more complex, not only in terms of size, but also

capabilities like reasoning, memory, multimodality, you may need tens of thousands or hundreds of thousands of these. What binds all of this together, what glues all of this together is the network, right? That's why I'd say network is the computer for AI infrastructure. Yeah. So for AI infrastructure, you're solving a fundamentally different problem. It's a distributed computing problem as opposed to a virtualization problem. So this is why organizations have to think very differently when they build this infrastructure and also think about differently when they're operating, optimizing or scaling this infrastructure. Yeah.

I think it's difficult for a lot of people to understand the scale and complexity of the network in an AI data center. People understand it like, okay, I need GPUs. They're really expensive. Invidia is the clear market leader in GPUs. They may understand the role of things like high buffer memory and the importance of that, or at least the supply chain constraint on it.

But I don't think a lot of people really appreciate the complexity of the network in a large GPU cluster buildout. Can you explain what that looks like, typically? Because you work with a lot of different customers at pretty large scale. Yeah. So when you're talking about scale or complexity, when you're building this AI infrastructure,

especially networking, you really have about in three different domains or three different vectors. The first is what we call the scale up domain. This is where you start building the rack in a data center. And in this case, in scale up domain, all of the XPUs are directly connected with each other. The network is actually pretty simple because they're all one hop away.

But what's not... We're talking about like in an invidia world, we're talking about like NVLink here, right? In the invidia world is NVLink, but I do believe that Ethernet will become the de facto standard for this kind of scale up. But really, you talked about the HBMs. They play a very important role because these GPUs are trying to access each other's memories. And today, we are talking about they have a memory bandwidth about 40 terabits in the next 12 months. We're talking about 100 terabits. Even if they're communicating 25% of the time, you're talking about,

about a network bandwidth of 10 terabits out of a single XPU moving to 25 terabits very soon. So think of this, our traditional data center where we still talk about NICs, which are 25 and 50 gig Nix and 25 terabytes coming out of an XPU. In this case, you have to deal with this very, very high bandwidth. But it's not only very high bandwidth. It's also you need efficiency because the network is simple. you don't want to burden this with a very, very complex large header, you can be more efficient.

Also, reliability, because these are memory transactions, right? That becomes very, very important. And last but not least, latency is extremely important. So if you just look at it, these are all the things you need to think about when you're building scale up. And then you can go to the next step, right? How do I now go beyond the rack? This is where you start connecting the racks together. And that's the scale out.

And over here, the top of point thing for people right now is, how do I keep this network to a two-tier topology? Because as soon as you go to three tiers, in scale out, the two main problems are load balancing and congestion. They become difficult to manage. And then, of course, if you go to the three tiers, you're using more optics, you're using more power, you have higher latency.

Just everybody understands that when we talk about tiers, we're talking about one tier of switches that connect servers together and then another tier that might connect multiple racks to each other, and then another tier that aggregates that again because of the scale of the network. Correct, correct. Or some of your audience may know about the leaf and the spiked. Or you go to a super spint. So you want to keep your two tiers.

And lastly, I think the latest what's happening is, like, if you run out of power and space in a data center, because if it's 10 megawatt data center, you can house around 6,000 XPUs. Now the cluster has to go across data centers. That is scale across. And when you're scaling across, you need to be able to keep a lossless fabric,

100 kilometers apart, keep the network utilization high, but also secure it. Because now you're going outside your data center, so you need this line rate encryption comes across. And you need a big pipe that moves packets in and out of that data center pretty quickly, right? So once you're deploying this,

you have to look at scale in this three different vectors. and optimize these three different vectors when you're building this infrastructure. Yeah. And we're talking about large scale, right? So on the typical switch that people are using in these data centers, how many ports are we talking about on a GPU server? When I think about n number of GPUs in a server,

what's the multiplication factor on network links and all the components that go into it? Yes, I mean, if you look at where we are today, There is 800 gig coming out of a single XPU, for example, for a scale-out connectivity, moving to 1.6 terabit. So we are the only ones shipping this 100-terabit switch, which is called the Tom Hoc 6. It's the first industry. It's the only one in production, the only one shipping in volume. And it can support 64 ports of 1.6 terabits.

Or you could actually have 128 ports of 800 gig with some breakout. you can build this because a lot of the data centers are now going with a liquid cool architecture and have a 2RU liquid cool design for this or you could have an air cool because power is, it's now very, very hard to build these air cool systems, but we have been able to keep this power footprint very low. So people are also able to build like 3RU, 4 RU air cool systems with us. So I start with a facility that has to be. has to have power. I got to work on the power distribution within that data center. I got to work on the cooling model within that data center. I then go buy a bunch of GPUs. They get racked in a scale up network. I've got multiple racks. They need to be all connected together in a scale out network. I need data going in and out of that data center, probably connecting with other data centers. I need it to scale across. I got all that. I'm getting switches with high performance silicon from broadcom. But there's software, right, that has to make all this stuff work? Yeah, no, absolutely. It's all layers of the stack. You will have software. If you're having Nick, you will have the firmware, the drivers. If you are running the switch infrastructure, you'll need to have software. And like we talked about, the software, you have options. It can be disaggregated software. It can be open source like Sonic. It can be an OEM kind of operating systems that's running on top. And then you also need to be able to manage this infrastructure in partnership with you. This is that you

you guys do very well, which is how you orchestrate all of this, but also monitor all of this. Because what is happening is if you are tuning this network or this infrastructure, where you're trying to say, look, I want to tune the load balancing mechanism to get some optimization. It cannot be very manual. You need to be able to do this very quickly. And then you need to be able to observe whether the action that you took is it having the desired result on the infrastructure. Yeah, that's why we put open observability into the product.

Yeah. Yeah, soft fit is key to make this book. Yeah. Yep. To really get optimal performance, and we're going to talk more about performance here in a minute and sort of its impact on the network performance and its impact on GPU utilization and token throughput. But there's kind of a magic config for this. You talked about performance.

You talked about a lossless fabric. Is it common knowledge out there on how to do the software configuration to get maximum performance and reliability and security out of this AI network? There are definitely guidelines. For example, we have a lot of experience in terms of building these very, very large clusters. This is where we have learned what kind of optimizations, even how the feature enhancements that have been needed that now you're seeing a new silicon. So there is absolutely guidelines that are available to do this.

How do you do the right load balancing? How do you do congestion control? How do you maximize the reliability of the infrastructure? but what happens is not all networks are the same. The workloads can be different. So there is always this room for optimization that is required. Right. And you need to have the talent.

If you're going to go build an AI cluster, you've got to have the network engineering talent to understand how to not just install everything, plug it in, but configure the software. So you're getting optimal performance and observability to improve that performance, et cetera. Absolutely. So you need that expertise. and for AI infrastructure, you probably need this beyond what has happened in the past.

Because, for example, the Nick is a very important part of this infrastructure. Nick was seen server-relevant. It's more of a networking element in the AI infrastructure. So how the host is behaving, how it's interacting with the network itself. And even actually a good understanding of the traffic patterns for when you're doing training or your inference, understanding the XPUs, the good thing is, anybody who is operating, absolutely. You need the expertise. But for the talent, there is a tremendous opportunity to learn this stack end to end, because that is how you can really build the most

optimal infrastructure. Yep. So there's an organization called the Open Compute Project. You're a member, we're a member, the sonic network operating system that you were talking about kind of came out of OCP. Hedgehog and one of your hardware partners, Celestica and one of our customers, Last year at the Global Summit, we presented industry-leading AI network performance on your Tomahawk 5 chipset, so an 800-gig network, where we were getting really, really good bandwidth and token throughput. That was independently tested and reported by semi-analysis for their cluster max rating system. And then that led to the OCP community asking us, hey, please publish a reference architecture so that everybody who's part of the community

can get something that out of the box delivers that same kind of performance. So we did that at the Spring OCP show. So here's my question for you, reference architectures. How should players in this AI infrastructure market think about reference architectures, where do they add value,

where are they useful, and where are they not? I think reference architectures are critical for this AI infrastructure, and I'll tell you why. There is a misperception out there in the market created by a few. Look, if you're buying an XPU from a certain vendor, you need the entire stack from the same vendor.

You need the networking. You need the nix. You need the routers. You need the switches. You need the optics. But the reality is far from that. When hyperscalers are building this infrastructure, yes, they will take the best of breed XPU.

But then they want to build their own network infrastructure that gives them the most optimal performance. They're cost conscious. They want to have choice in terms of where the cables, optics, and all of these things come from. And this is where the reference architecture is very important because, okay, hyper-scalers can do this, but all of the other customers want to take this route.

And they want to go down this open route. But when you go down the open route, you have to give customers the choice. We believe we build the best network, but there are others players out there who can do the same. And the reference architecture gives customers the guidelines, which means that if you are going with an open architecture, this is how you can configure, optimize.

Here are the choices that you have. And it also gives them the confidence, right? That if I go down this route, it's validated. So I think they are essential moving forward. And I want to thank you for the role that you have played in doing it. I think you guys have done a fantastic job. And I also, big fan of OCP, where they have helped publish this design guide and they continue to more and more work on this front.

Yeah, I think it's great. It's really a pretty fantastic organization. So look, there's a lot of pressure right now for pretty much every business on the planet to have an AI strategy. And a lot of people are using public models and they're getting, in a lot of cases, great results. But at the end of the day, they get a bill for tokens. So how should organizations balance the demand for, hey, make AI happen fast with the economic realities of public models, public tokens, maybe you run in a private model and building AI infrastructure.

How do you sort of optimize efficiency and control costs? Yeah, efficiency and controlling cost. I think that is top of mind. And it has been top of mind for the last four or five years, starting with training into inference like you just articulated. To give you an example, this is four years ago, when Meta came to us and the whole OCP and other forums and said, look, I am running these recommendation models and trying to train these recommendation models and my GPUs are sitting idle

up to 57% of the time and it is because of the network. That just tells you the importance of something like the network infrastructure when you're building distributed inference or training clusters because networking is only 20% of the spend when we're building the infrastructure. but it can get in the way of the most vital optimizations that you need. 20% of the CAPEX. OpX, it can have a pretty huge impact, right? Exactly. So to your point, if I can now increase the utilization,

which you've been able to do by 15 to 20% of this compute, the network has essentially paid for itself. Yeah. Just like within a month or two, yeah. Exactly. Right. So, yeah. I think, I don't think everybody really understands this.

It is probably worth talking about it, a little bit, but when you've got an AI workload, it never runs on just one XPU, typically. There's typically groups of XPUs. They need to be able to share memory. They're doing that over the network with the goal of generating tokens. And if the network isn't operating at peak efficiency, you don't get as many tokens. The model flops utilization of that XPU goes down, right? I think is what you're describing.

Absolutely. You're describing the inference use case. You have the refill phase, which is a little more compute heavy, and then you have the decode phase. And in the decode phase, it's memory intensive, it's all the communication and network. And you need to be able to generate these tokens as fast as possible with the lowest latency. If you don't have the right infrastructure, the right network in place, you are losing against your competition that's out there. And I think this will become even more important as agentic takes off more and more. I hear a lot of agentic, agentic a lot of.

But the thing is, when you talk about agentic, you are telling, look, I've, I, I want to book a flight to Paris for my next vacation and you have given this job to the agent. The amount of context that is coming in, it's huge. And now you're thinking of tens or hundreds of agents doing this for individuals. What we see right now from a requirement's perspective, it is just going to just exponentially become more and more important that you have the right infrastructure, network infrastructure in place to get the right option. Yeah. I think Jensen did a great job in his GTC keynote, just talking about really a new global economic economy for the world, which is tokens per watt. You want to maximize that. And you can't if you don't have a high performance network and you don't have observability on it and you're not automating response to issues in the network that if you don't resolve them, it drives the token throughput down. And it ends up cost you.

you a lot of money, like a lot of money that I think a lot of people just don't quite understand yet. So yeah, I think it's super important. Tell me this, because there is, look, networking has been around for a long time. Different equipment manufacturers have developed their own operating systems. You accurately noted that most people are going to want to buy infrastructure from multiple vendors. They have to just because of supply chain availability in a lot of cases. So if you end up in a scenario where you have equipment for multiple vendors with multiple network operating systems, all trying to work together to create these GPU clusters to do everything we've been talking about.

What happens? Also, with Sonic as well, when you start getting this fragmentation with different versions of Sonic, what happens to the operational complexity? How do you manage all that? First of all, I would just say like this openness and interoperability is fundamental to the success. of AI. There will be like hundreds of gigawatts and millions of XPUs that will be coming online over the next four to five years. And it is not possible

for just one vendor to do all of this. You need an ecosystem that's an open ecosystems. And this is where customers are realizing. And this is why Ethernet is becoming the uncontested standard where you build out this infrastructure. And this is where we have worked

with you, with the other partners, in the Ultra Ethernet Consortium and OCP, to replace these proprietary silos like Infineband with high-performance, open standards, Ethernet. So if you have different options, it's actually there is one consistent element. You have Ethernet as a standard protocol, no matter where you're buying from.

And at the end of the day, you have the operating systems like you talked about, but they will have APIs. And for example, somebody like your team, Hedgehog, can hook onto this open AIs and open APIs and provide a very consistent view across the infrastructure. But if you have multiple proprietary technologies doing this, then it becomes very, very complex to do this.

Yeah. We talk a lot about part of Hedgehog's value is providing a common abstraction. It's cloud native, operates like public cloud. You can operate it with the DevOps team. And for equipment that's coming from multiple vendors, and we just abstract all that complexity away. So it's a big part of what we do. You mentioned Ultra Ethernet Consortium,

and Ethernet, which is really the networking standard that everybody uses for everything, including this call right now. Can you talk a little bit about the Ultra Ethernet Consortium, what it is and what it's doing and what we should expect from it in the future? The Ultra Ether Consortium, I think, about four years ago, we took a stand at that point of time

that Ethernet should be the way to build at least the scale-out infrastructure for AI. And at that point of time, there were proprietary options. People thought that was the best way to do this. But after that, slowly, people, when they started realizing that, look, I need, I can't be locked into proprietary solutions. We need openness. Ultra Ethernet Consortium was formed. I believe there are now more than 200 members.

And it has played a very critical role. Because, look, it's the innovation cannot be just done by, one company and one person. It's a collective responsibility. And one good example on the Ultra-Eythens consortium is, see, now you can see the cluster sizes are moving to 100,000 to a million maybe in a few years. But ultra-eithen consortium, just to give you one example, one problem that was identified is the underlying protocol that is used to build this infrastructure is RDMA. It is more than two decades old, right? It cannot support multi-pathing. It cannot do out of replacement, cannot do selective retransmites, old congestion control mechanism does not have a lot

of telemetry. It needs to be modernized. And they came up with the spec, the 1.0 spec. And that was the collective wisdom of the consortium. And now it's available. From a customer's perspective, like we have implemented this, for example, the features that are required in the spec in our latest Nick, we call it the Thor Ultra. And from a customer perspective, the benefit is, now you can get consistency across the board. So I think that's the role that one example on how they're on the scale up side, there are capabilities like the link level retries and the credit based flow control. And they have been able to standardize those, standardize those for everybody.

You know, at the latest ESUN kind of guideline that has come from OCP, it is taking those two recommendations. So again, now anybody who's going to ask for a switch, they will ask for a switch. for LLR and CBFC, at least it was standardized in UEC. So a very, very important role. And as we move forward, as we scale this infrastructure, as we build out more and more, I think this innovation will come out,

but it will be standardized. They'll make sure it's standard, make sure that people are interoperating on this part. Yeah. So just recap all that, RDMA, remote direct memory access, sort of older network technology, you can think of it as AI network performance 1.0, ultra-ethernet consortium now establishing open standards for interoperability so you get

AI network performance 2.0. Simple way to think about it. That's a good view of phrasing this. Okay. Thank you. I just try to summarize everybody gets it. Okay. So, hey, look, this podcast is called the AI hedge. For people thinking about financing and building and operating,

a new or AI cluster, what do you think are the primary risks that they should be thinking about? We talked a little bit about performance. What else? I would say the number one risk is losing your architectural sovereignty. Because if you lose,

there are a lot of solutions out there. Like I said, look, they will force you to buy the entire stack. But losing your architectural sovereignty over your AI infrastructure means your entire business strategy is tethered. You're locked in, right? You get locked into a single vendor.

Is that, I don't know what you're describing? To that vendor's pricing and proprietary constraints, right? I mean, these closed systems may give you a tiny bit of short-term optimization, but the real trade-off is just losing the strategic flexibility you need in a very, very unpredictable future. So I think my message would be own your architecture, own your network, on your future. Hyperscalers have been doing this for a long time. That's why the Open Compute Project exists.

And that the challenge now is to enable enterprises and neoclouds to network like hypers with the same core principles of owning your network, control your destiny. Absolutely. Broadcom as a company has a lot of stuff going on. What roadmaps at Broadcom are you most excited about? Generally speaking, I am extremely excited about our roadmap. But I think what really excites me the most is just solving the customer problems that we are introducing in this roadmap. The thing is, there is no shortage of problems to solve.

That's exciting. But you're removing these hurdles for all these customers as they try to drive towards superintelligence. One good example would be like MRC, multipath reliable connection. It's a new protocol. And two years ago, again, Open AI could see that they need to be. go to cluster sizes which are very, very large. So they need to that two-tier, three-tier architecture that you were talking about earlier, right? Correct.

So how do we keep this architecture to two tiers, how we move to a multi-plane architecture, that was their requirement. Microsoft was the one who had to implement this in the data centers. And they came to us and other vendors on how to solve this problem. This is where we actually took a lot of learnings from Ultra-Eathenet Consortium. He said, look, let's not try to recreate things. Take the learning from there, but we need to move to a more centralized network architecture, which is SRV6-based to give more predictability, to ensure we minimize congestion,

concepts like packet trimming came about. So I think solving these kind of problems is what is exciting for me, exciting for the team here. And also the collaboration that's happening. It's also good to work across the industry to, to solve. solve these problems because again, I say this, you can be in your own room, you can try to solve this, but this collective wisdom that comes together. In this case, for example, you had an LLM player who was thinking LLMs. You had a company who was extremely good at running very, very large data centers. And then you have somebody like us and like somebody like Tia and somebody like

EMD and Intel who are bringing their own expertise. So I think that's what's exciting. Yeah. I mean, when you see that industry collaboration on a really big, really hard problem. It's pretty exciting. I know you're a networking expert, you're deep in networking, but if you think about the whole AI infrastructure stack and the kinds of intelligence that people are expecting in the future, what areas of that AI infrastructure stack do you think are most poised for the biggest transformation over the next three to five years? first thing, it's more network-centric, but I think what will happen is

for scale out and scale across, everybody has come to a consensus that Ethernet is the way to go forward. Now, scale up is where some of the discussion is. I think that will become the de facto standard as well. But I think in this case, what people are really, as inference becomes bigger, people want to have the scale-up domain

which is not confined to Iraq. They want to go across. And they want the scale-up domain which is 5, 12, 1,000 units. And I think you have solved most of these problems from an Ethernet perspective, whether it's latency, whether it's bandwidth, whether it's reliability, whether it's efficiency.

But now imagine if these racks are far apart. All of this connectivity is over copper, but how do you bring in something which gives you the reach beyond copper, but without compromising the power, the cost, the reliability benefits of course? copper. So I think this one technology that is something called this optical compute interconnect OCI, that MSA is out there. So innovation in this space, something along this lines, is something to watch out for. And the other aspect you talked about, which is inferences taking

off. It's becoming bigger. Agenic is becoming bigger. I think there is a lot of room for disruption and innovation in this space. Because like we discussed, there is a pre-fill stage and there is a decode phase. And the decode phase, it's very memory intensive, it's communication. Of course, there is a network innovation, but all of this innovation that is needed on how do you deal with all of the storage in this space and how are able to get to the storage and get to a result as fast as possible? I think that's exciting.

There are problems that need to be solved. And I think we should be prepared to see a lot of innovation and disruption in the space. Yeah, for sure. Sure. Well, look, Hassan, this has been a pleasure as always. It's fascinating. Every time I talk to you, I learned something new. I want to thank you for your time. For listeners who want to learn more about Broadcom and what your team is doing, where should they go? First of all, again, Mark, thank you very much. I really enjoyed the conversation. Like every conversation we have, if you want to find out more, use AI. Absolutely. There you go. That's the right answer. I don't need to use. URL anymore. Just ask AI. We have a lot of good content on the Broadcom website because, yes, people do get over-vermed. I would say this is where we have the content, which is very broadcom-specific, but also we believe in openness. So like the stuff that you guys are doing,

from the reference architectures and everything else, I would also say follow the social media handle, the Broadcom social media handles, whether it's LinkedIn or Twitter, I think you will find very useful information on a daily basis. Great. All right. Well, thanks a lot, Hassan. We're still having a lot of fun on this journey, and I feel like it's just beginning still.

So I'm really happy to have you as a partner, and thank you. Dato, Mark, thank you for the partnership, and thank you for having me here.