Utilizing Tech - Season 7: AI Data Infrastructure Presented by Solidigm - 13: Will Nvidia Dominate the Market for AI Hardware

Episode Date: November 17, 2020

Matt Bryson of Wedbush securities joins Stephen Foskett for a discussion of AI hardware companies, focusing on the biggest player, Nvidia. Stephen and Matt start with a look at Nvidia: Just how big is Nvidia in the enterprise AI market? Then we turn to other major player in this space, Intel, which is strong in the inference market with their Xeon processors but obviously wants a bigger piece of the special-purpose processor market. AMD has had success in the cloud but doesn’t seem focused on the AI space. Then we look at the world of AI hardware startups. How will they compete with Nvidia, Intel, and AMD, when they just don’t have the same resources? Companies like BrainChip and Cerebras are trying to be more efficient and go after the gaps in the market rather than compete directly with Nvidia. Then there’s the crossover between AI and HPC, which is an opportunity for AMD, Tachyum, and others. We also see an opportunity for AI at the edge, which brings to mind companies like Apple and Huawei who are adding AI processing to chips used in client systems. We also need to consider companies like Amazon and Google that are creating their own AI solutions and Microsoft using GraphCore. But does AI live in the cloud or will next-generation hardware platforms like Liqid be more compelling? Finally we turn to the pending acquisition of Arm by Nvidia, and what that means if it goes through and if it doesn’t. Episode Hosts and Guests Matt Bryson, Senior Vice President at Wedbush Securities, can be found on LinkedIn at LinkedIn.com/in/Matt-Bryson-3105071/ Stephen Foskett, publisher of Gestalt IT and organizer of Tech Field Day. Find Stephen’s writing at GestaltIT.com and on Twitter at @SFoskett Date: 11/17/2020 Tags: @SFoskett, @Nvidia, @Intel, @AMD, @BrainChip_Inc, @CerebrasSystems, @Tachyum

Welcome to Utilizing AI, the podcast about enterprise applications for machine learning, deep learning, and other artificial intelligence topics. Each episode brings experts in enterprise infrastructure together to discuss applications of AI in today's data center. Today, we're discussing hardware, and specifically NVIDIA's position as the 800-pound gorilla of AI hardware. First, let's meet our guest. Hey, Stephen. This is Matt Bryson. I'm the VP of Research Covering Hardware over at Wedbush Securities. How's it going?

Great, thanks, and thanks for joining us. I'm Stephen Foskett, organizer of Tech Field Day and publisher of Gestalt IT. You can find me on Twitter at S Foskett. So Matt, you're one of my go-to guys when it comes to enterprise technology and specifically the sort of finance and M&A and all that kind of stuff that happens with these giant, giant companies. And certainly one of the companies that you must be following and focused on is NVIDIA. So let's start by just looking at them and this amazing position they've got in the enterprise AI space. I mean, how big is their position in the market? So that's really difficult to come up with a specific figure there. If you look at their

numbers, they're doing over a billion dollars quarter at this point the the problem is there's really not good published data around what the entirety of the market looks like so it's clear they're doing a ton of revenue it's less clear how much other people are picking up yeah and and it seems um certainly that they're the the you're the kind of company on everyone's lips. But in a way, you know, it almost reminds me of the position of like an Intel or something where, yeah, they make a lot of money and they sell a ton of processors, but are they really dominant if you look at all the other players in the market? So I think they are. If you think about FPGAs, which have been brought up as a competitor in the data center,

Xilinx is only selling $150, $160 million worth of parts a quarter. And that's for multiple applications. You compare that with NVIDIA. Again, they're over a billion dollars a quarter. I just don't think anyone has the scope that they do. Maybe Intel is somewhere up there, but it's so hidden in their numbers, it's hard to tell. Yeah. And certainly, Intel is a big player as well.

But I mean, Nvidia, they just keep hitting on it every time. They've got their GTC conference. Frankly, they talk about the AI market more than they talk about the gaming market. And I imagine that, you know, gaming might represent a bigger market for them. And in fact, embedded might represent a bigger market than, you know, sort of enterprise AI at this point. But yet, it seems like that's what NVIDIA is really hoping will be the, I don't know, the horse that will pull their wagon into the future. So I think if you look at their revenue forecast, they talk about a total available market of $100 billion in 2024. The bulk of that, you're talking 80 plus is ai and their view is that the entirety of the data center market edge applications um that effectively ai can fit almost every problem and i don't know how accurate that is but when you take that from a wall street perspective if i tell you that looking forward 80 of my revenue is going to come from this area, then that's

the bulk of the company's value when Wall Street is looking at it. And I think that that's the kind of thing that folks like me don't always see because I'm really focused on sort of the cool gear. And yet we don't always know like the reality of some of these, you know, companies positions in the space. You know, Intel is obviously another huge, huge competitor in this space, or at least they'd like to be. We were at the, you know, the Intel Xeon scalable launch last year. They focused a lot on AI. Ever since then, you know, they've been talking about it more and more. And in fact, you know, Intel is going to be a presenter at our upcoming AI Field Day event.

And they've got a lot to say. And it's not just processors either. It's not just like, oh, you know, here's the next generation graphics cores that can do, you know, AI. I mean, they're talking about using Xeon for it. They're also talking about accelerating workloads with Optane and so on. But let's focus on the AI processor market. You know, does Intel even have a blip in the enterprise AI space right now? Or is this sort of an optimistic suggestion that they'll be able to be a player there? So I think that if you look at the inference market traditionally inference has mostly been run on Intel processors now if you believe when videos said that

over the last couple years they've managed to shift 50% of that market to Nvidia graphics processors and they believe that they'll have over 90% of the market in another couple of years. So Intel has this strong CPU story with Xeon, but they also, of course, have tried to do general purpose or special purpose processors from Xeon Phi into a whole bunch of other things. So Matt, what can you tell us about that? So I think with Xeon Phi, they ran into the same problems that they've had on the CPU side. That was supposed to be a 10 nanometer process and they just never got the

process down. So they started selling into HPC as an accelerator and it went nowhere. They killed the product. I think they fractured some relationships in doing that. And now they're on their second startup. I think what that tells you is that Intel sees that they need something that's more than a Xeon processor to capture this market, but they've struggled a bit to get that architecture

in the right place. Absolutely. And it seems as well, I mean, from a very, very technical perspective, it seems like maybe Intel misjudged the AI market because they, you know, they were going for these, you know, massive, you know, floating point units and yet the AR market has seemed to coalesce more around even smaller and smaller, like 8-bit matrices and things like that, that Intel, I think they missed it. I think they made a mistake. what other companies are working in the space? You know, do we see AMD with general purpose GPUs as accelerators? I think AMD is trying to get there. But if you look where they've had success so far, it's selling standard graphics processors in the cloud to support something like Google's gaming effort.

And then to some extent on the HPC side with accelerator. So they've done a lot of work around integrating workloads between the CPU and the GPU that is somewhat kind of unique in their capability to do that. I think it plays very well on the HPC side, but it doesn't seem to migrate into AI. And I just don't think they have the software infrastructure right now to compete with NVIDIA. So the next area that I'm thinking about is these startups. Now, I've spent a lot of time talking to a lot of startups in this space, from the big guys to, you know, the small ones.

You know, I, in the last week, have talked to, you know, Tacium and BrainChip about their alternative silicon, let's say. I guess before we talk about any specifics, like how do smaller players get into this market since they really can't challenge, you know, the development muscle of somebody like an NVIDIA or an Intel? So for the startups, I think they have to develop something different. If they try to go toe-to-toe with NVIDIA in particular, with the same type of hardware, the same type of software platform, they just don't have the money to replicate what NVIDIA has done, and they don't have the time that they would need to get there.

They have to come up with something that's unique. And so what I've run into is companies that are trying to deal with the data differently. So let's say deal with fewer data points or find ways to sort through the data before processing. So you can deal with a smaller data set. You can do whatever you're doing in a more expedient manner

and get a faster result than NVIDIA would give you. And I think that's what the startups have to do. They have to find something different and unique that they can offer that NVIDIA can't. Yeah, exactly. So for example, like BrainChip, when I was talking to them, they were saying that they're going down to as low as four bit precision. And because of that, they're claiming, you know, just massively less data to deal with. And the fact that they can

process stuff at lower precision means that they don't need to have as many transistors, they can use a bigger process node, a cheaper chip, et cetera, a lower powered chip as well. So that's one approach. And then you've got the absolutely astonishing Cerebrus CPU. I mean, I know that you know a little bit about that guy. Yeah, no, they have a huge chip. But again, I think what they would tell you

is they were working in a similar fashion in that they are trying to find ways going to be divided between NVIDIA for certain tasks, in particular tasks that just require a lot of muscle, and that there'll be a startup who emerges who does something better, who takes a different portion of the market. Yeah, absolutely. There seems to be, for example, like a mobile opportunity, you know, for mobile IP, there seems to be an opportunity in HPC. And that's actually another angle, too, that, you know, you see these companies like Cerberus

that are, you know, there really is sort of a crossover between the HPC and the AI space? There certainly is, though. I think that the use cases are somewhat different. And so we were talking about AMD a bit earlier. AMD's had some success with the HPC space, just again, because they're able to work with their GPUs and CPUs in tandem. I think again there are different use cases and so certainly you can provide different approaches that will allow you to get portions of the market. I think the interesting piece is, is there someone else who can go after the bulk of the market in the way that NVIDIA has done between test and now inference and kind of take the mainstream market. Yeah, I think that's going to be the challenge to try to find.

When the elephants dance, the grass gets trampled and try to figure out a way to not get trampled from the AMD and Intel and Nvidia just fighting it out for the big money in the market. But they could find, that was the interesting thing as well is that I'm talking to these small startups and it really does remind me of the ARM versus Intel world in that, you know, Intel is unarguably the CPU leader, but yet there are way more ARM chips out there and they're everywhere. And, you know, I think that that's the idea with some of these startups as well, is they're thinking, you know, hey, if we can get our stuff in like everything, then, you know, it doesn't matter that NVIDIA dominates the data center. Yeah, and I think there's also an argument to be made out there that NVIDIA is a very power-hungry solution. So if you can come up with something that's low power, even if it's meant to live at the edge, and there's certainly, there should be demand for AI at the edge.

One of the problems with AI is it needs to consume data and data movement is expensive. And so if you can come up with a solution that can deal with the problems of AI at the edge, it very well may also work for AI in the data center. And that's, I mean, that's what we're seeing with processing at ARM right now, is that you are finally seeing the beginnings of large, both data centers and HPC installations that are utilizing ARM rather than x86. And also, they're adding specific AI instructions to these CPUs.

I know that ARM has been talking about that. Apple already has that. I mean, one of the signatures of the current, in the last couple of generations of Apple processors has been that they have that neural engine in there for the edge. And is that a threat to these smaller companies? I mean, is there a threat that somebody like an ARM or Apple might just bake in enough to handle the edge AI, and then they don't need a special purpose processor?

I think certainly, so you mentioned Apple. Huawei has done the same thing with their chip, though obviously they have struggles with what the US is doing to their ability to import semiconductor material at this point. But I think there's certainly an argument to be made that there are some large companies at the edge who may eventually have ways to innovate such that they are able to tap in or create that low power market but then you also have the cloud companies themselves right and you have Amazon and you have Google who are working on their own AI solutions and oftentimes it seems like they're using again architectures architectures that might take advantage of or might be able to live with lower power and look different than an NVIDIA part, for instance. Yeah, I was actually going to talk about that because that, I think, is perhaps the ultimate unknown. And, you know, if, if AI can live in the cloud, and if the cloud providers can see an opportunity to develop their own AI solutions, whether they

leverage, you know, an off the shelf chip or somebody else's IP, or if they are developing their own, as you mentioned, you know, we've heard of, you know, Amazon and Google working on things like that, and deploying things like that. Does that throw this whole thing on its side when you've got those kind of companies with those kind of resources? So I think if you look at AI, there's a couple things going on. There's a division between the cloud and the enterprise where, again, data locality matters. And so I think that in general, the cloud has done well with solutions that live on the cloud, where the data lives on the cloud.

And so you can craft AI solutions that that manipulate that data um i i think what the struggle is and if you look at nvidia's numbers they'll tell you this 50 of the revenues are enterprise and if you talk to the hardware vendors like an inspir like a liquid like a super micro they will tell you that one of the few areas where they see people moving data and applications off the cloud is around AI because it's so expensive to run AI applications on the cloud. So I think that you've got a struggle in that does AI live in the cloud? And then you've also got a struggle in that, how good are these companies at creating silicon solutions? And so if you look at the storage side of the fence, for instance, there

was a large push for Microsoft, Amazon, Google, to build their own SSDs, their own SSD controllers. And I think what you're seeing now is to a large extent, those cloud players are migrating back towards standard SSDs. And so in this case, I think that every cloud player wants to have an alternative to Nvidia. Like they don't want a market monopoly. The question is, how do they get there?

Are you better off leveraging a Graph Core? And so Microsoft, for instance, has put Graph Core in Azure, letting them do the work, raise the investment and using them as an alternative, or are you better off being Google or Amazon, building your own solution? And then with Google, you've got an advantage in that you control TensorFlow, right? So can you build a solution that fits with your vision of the software moving forward?

Certainly. But I think it's one of those questions where, again, you end up in a situation where they're only able to cater to a portion of the market, not the entirety of the market. And because of that, you still need someone like an NVIDIA. And so, it could be a second or a third competitor, but it wouldn't be the dominant player. So, yeah, I mean, we've kind of come, you know, full circle here from, you know, the dominant player all the way around to all these different areas, different aspects of it. I guess maybe the question is wrong. You know, maybe the answer is that assuming that AI takes off, we're going to have AI processors in the data center. We're going to have them in the client and the edge.

We're going to have them in the cloud. And they're not necessarily going to all be the same processors, right? I mean, is that a reasonable way to look at it? I think that's a fair way to look at it if NVIDIA is not able to acquire ARM. So I think to your point earlier about could you have these devices at the edge that eventually maybe compete in the data center? I think that NVIDIA is looking forward and definitely seeing this market for edge compute and edge AI and thinking about how they could capture that market, but also thinking about what competitors could potentially do with Arm's architecture, right? Because Nvidia could certainly license Arm as IP.

They are as part of the deal, and then build their own device. I think when you think about them owning Arm instead of just licensing intellectual property, I think that tells you that they see this emergence of AI at the edge and they've got to think that it's better to own the potential competitor. Yeah, that's actually a really interesting point and one that I hadn't really thought all the way through in that, you know, maybe the end game here isn't, you know, to kind of corner the market on mobile CPUs. Maybe the end game here is to make sure that the NVIDIA solution is the standard AI solution, the standard

AI instruction set, I guess, if you will, from the data center to the HPC to the edge. And if that's the case, then, wow, this really is a pretty big play for them. Yeah, I think that NVIDIA is looking at the edge and they use their GPUs but I think that their vision is beyond their GPUs living at the edge it's beyond owning arm because I can take my graphics IP and I can add it to Arm's IP portfolio and sell it to anyone designing a processor for a handset. I think it's more that you're going to need ton of success at the edge because they offer a low power solution that fits very well at the edge. Yeah, absolutely. And I think that that's, well, I guess that's maybe the answer to the question. You know, I mean, is, you know, if NVIDIA is going to dominate the AI space,

then really the path to doing that would be this acquisition of ARM. And, of course, we don't know if this is going to happen. It's been announced, but, you know, there's a lot of, you know, approvals that still need to be made before this acquisition can happen. I guess to finish up one final thing, if the acquisition is derailed by regulators, you know, does that open the door to not just a competitor in the edge or low power space,

but perhaps even eroding NVIDIA's position as the dominant vendor of data center AI chips? I have certainly heard the thesis that there will be a play for a low power vendor in the data center. And that Hyperscale will take this low power solution, they will design their own software, and that will eventually be the NVIDIA killer. So there's a path there. Having said that, I don't know what the mechanism is that moves us down that path.

And remember that NVIDIA is paying, I believe almost a billion dollars to license Arm as part of the deal. That license doesn't go away. So they will certainly have a product set that fits into the space we're talking about. It's just if they were able to control ARM and impart work around their technology progression such that it fits NVIDIA's vision, then their solution would be that much more compelling and that much more difficult to displace. But I guess the answer is, I think NVIDIA is going there. It certainly makes it more likely that you end up with a competitor beating them to the punch if they don't control arm. Yeah, absolutely. And so it really, you know, I don't want to say

that it makes it a desperate situation, but it makes it a very important situation, you know, not just for everybody else in the market, but even for the dominant player in the market, NVIDIA, because, you know, it's still early days. And I guess, you know, to finish up from my perspective, it still is early days. And so there's every chance as well that something totally radically different could happen. And, you know, this could all look like, you know, arguments about, you know, the old x86 competitors, you know, back in the early 90s or something. I mean, you know, we don't know what this space is going to look like. Well, thank you so much for this conversation, Matt. It's really enjoyable to speak with you. Where can people connect with you and follow your thoughts on enterprise tech? Probably the best place to connect with me is LinkedIn. Just being in the finance world, my ability to use social media is somewhat curtailed.

Well, I really do appreciate it. And like I said, I always appreciate getting a chance to speak with you. And thank you everyone for listening to the Utilizing AI podcast. If you enjoyed this discussion, remember to subscribe, rate, and review the show in iTunes, since that really helps our visibility. And please do share this show with your friends. This podcast is brought to you by gestaltit.com, your home for IT coverage from across the enterprise. For show notes and more episodes, go to utilizing-ai.com or find us on Twitter at utilizing underscore AI. Thanks, and we'll see you next time.