Daybreak - The Microsoft vs Nvidia battle, powered by an army of Indian engineers

Episode Date: January 13, 2025

On the 25th of November 2024, just five days before the placement drive began at the Indian Institute of Technology, Madras, a new role was suddenly added to the student’s placement portal. Tech giant Microsoft was looking for a bunch of promising silicon hardware engineers. This was the first time Microsoft was hiring for this particular role. And it wasn’t just looking at the top tier IITs like IIT Madras. Students at IIT-ISM Dhanbad in Jharkhand, saw the same exact thing happen. Why, you may ask? One word. Nvidia. Until now, companies like Microsoft, Amazon and Google have been the most direct beneficiaries of the AI revolution. But the Nvidia monopoly in the AI chip market is preventing these tech companies from making as much money as possible. Microsoft wants to change that. That’s exactly where a team of freshly minted hardware engineers comes in. The company is hiring graduates who know hardware programming to test and design its own chips. Tune in. Tell us what you thought of this episode. You can text us your feedback on WhatsApp at [phone]Daybreak is produced from the newsroom of The Ken, India’s first subscriber-only business news platform. Subscribe for more exclusive, deeply-reported, and analytical business stories.

Hi, this is Rohan Dharma Kumar. If you've heard any of the Ken's podcasts, you've probably heard me, my interruptions, my analogies, and my contrarian takes on most topics. And you might rightly be wondering why am I interrupting this episode too? It's for a special announcement. For the last few months, I and Sita Ramon Ganeshan, my colleague and the Ken's deputy editor, have been working on an ambitious new podcast. It's called Intermission.

We want to tell the same. secret sauce stories of India's greatest companies. Stories of how they were born, how they fought to survive, how they build their organizations and culture, how they manage to innovate and thrive over decades, and most importantly, how they're poised today. To do that, Sita and I have been reading books, poring over reports, going through financial statements, digging up archives, and talking to dozens of people. And if that wasn't enough, we also decided to throw in video into.

to the mix. Yes, you heard that right. Intermission has also had to find its footing in the world of multi-camera shoots in professional studios, laborious editing, and extensive post-production. Sita and I are still reeling from the intensity of our first studio recording. Intermission launches on March 23rd. To get an alert, as soon as we release our first episode, please follow Intermission on Spotify and Apple Podcast. or subscribe to the Ken's YouTube channel. You can find all of the links at the ken.com slash I am. With that, back to your episode.

On the 25th of November 2024, just five days before the placement drive began at IIT Madras, a new role was suddenly added to the student's placement portal. Tech giant Microsoft was looking for a bunch of promising silicon hardware engineers. Now, this was the first time Microsoft was hiring for this particular role. And it wasn't just looking at the top-tier IITs like IIT Madras. Students at IIT-ISM Dhanbad in Jarkhand saw the exact same thing happen. Why, you may ask?

Well, one word, NVIDIA. You see, Microsoft and pretty much any tech company worth its salt, is pushing hard to make its mark in the AI and cloud space. That's because the demand for AI services is surreal at the moment. Until now, companies like Microsoft, Amazon and Google have been the most. most direct beneficiaries of the AI revolution. That's because their cloud services are the easiest way for enterprises to access AI solutions. But while companies like Microsoft have seen revenues for their intelligent cloud division

jump by 40% or more in just about two years, it is the Nvidia monopoly that is preventing these tech companies from making as much money as possible. The Ken reporter Abhiramiji explains why. So the thing to understand here is that you, have two kinds of hardware that are used for computing here. One is a CPU, which is the general purpose processor that you find in personal devices like your computer or your phone or anything like that. And then you have GPUs which until maybe a few years ago was a processor that only gamers

knew of or people who worked in specialized video applications but are now have exploded all over the scene because of the rise of AI. From a business perspective, what just means is that CPUs are very widely available in the market. There are many different companies producing CPUs, so you kind of have a lot of choice as to what CPU to pick. Whereas with GPUs, the market leader by a mile is Nvidia. So this essentially means that if a person or a company or like any kind of entity is looking for a suitable hardware that can run its AI processes, NVDA is really the only player it can turn to, which Nvidia is also very well aware of, which means that you kind of have a situation where Nvidia can adjust prices as much as it wants. Quite naturally, tech companies of all shapes and sizes resent the Nvidia monopoly.

But it's only big tech, companies like Microsoft that can actually do anything about it. And that's exactly where a team of freshly minted hardware engineers comes in. You see, the company is hiring graduates who know hardware programming. Now, that is a skill that major chipmakers like Nvidia usually look for in their new hires. At Microsoft, these recruits would be tasked with testing. testing, verifying and designing chips that the company would then use for Azure, its cloud computing servers. You see, the company has been going all out. Since late 2023, it has already released two chips, the Maya 100 and Cobalt 100.

The hired engineers will join a two-year-old high-level team called Silicon Cloud Hardware and Infrastructure Engineering, or Shai. And they have their work cut out for them, because now the stakes are even higher. You see, NVIDIA is done being just everyone's preferred chipmaker for AI workloads. It's also planning to launch a development environment of its own to write AI programs. And that could be bad news for intermediaries like Microsoft's Azure. Welcome to Daybreak, a business podcast from the Ken. I'm your host Rahil Filippos and I'll be joining my colleagues Nikta Sharma every day of the week to bring you one business story that is worth understanding and worth your time.

Today is Tuesday, the 13th of January. It isn't particularly surprising that Microsoft is looking for chip design talent here in India. After all, one in five people in the space across the world come from here, which is why the biggest chip makers, companies like Texas Instruments, Intel and AMD, all hire from engineering colleges here in India to work on their outsourced chip design tasks. But the catch here is that while there's a lot of talent, What India lacks is people who can ideate and design products. That's what Arjun Mennon, co-founder and chief engineer at Incor Semiconductors,

which is a chip design company, told Abhirami. Now, the top students who are skilled enough to design chips generally end up going abroad. So tech companies usually have to look beyond that. The advantage that Microsoft has is that people would most certainly choose it over a startup. You see, yes, India is a hub of electronics design, automation, and chip design, which is precisely why global chip giants like Intel, Nvidia and AMD, have their R&D and design centers in the country. But Indian firms generally specialize in low-rung areas like system-on-chip design.

Abirami spoke to some of the students who sat for the interview at IIT Madras. They said that during the interview, the company prioritized candidates who knew Verylok, which is a hardware programming language. They were told they would be working on Microsoft's internal stats. since this would not be a product-based role. According to Menon, freshers are more likely to be hired for verification work. Now, that's the process of stress testing

and checking whether chips work according to specifications. You see, this is all part of Microsoft's big project to build cheaper and better alternatives to Nvidia's offerings. That, according to a former Microsoft employee, is something enterprises will most certainly start looking for once the gen AI hype starts subsiding a little bit. And that could happen as early as in the next nine months. So where do things stand for Microsoft right now?

More on that in the next segment. When Microsoft revealed its Maya 100 specifications and features last year at Stanford University's Hot Chips 2024 event, what really stood out was that the company's chips were being built on older tech than Nvidia's and at a lower cost. What many tech commentators were quick to point, out was that Microsoft seemed to be able to make these chips a lot less expensive than

Nvidia's GPUs. But the thing is, Microsoft is still in the early stages of building their chips. Meanwhile, Amazon and Google have been at it for a while. Amazon, in fact, even has its own chip. It's called Trinium. But according to Meinen, it isn't an apples to apples comparison between the three companies. You see, numbers wise, Amazon and Microsoft are almost neck and neck in the cloud segment. Their operating incomes in the September 2024 quarter both surpassed $10 billion. In comparison, Google's cloud operating income for the quarter was only about one-fifth of that figure. Amazon has also announced the second generation of Traneon around the same time as Microsoft's Maya chip. But the thing is, the same former Microsoft employee Abhirami spoke to,

said there's only so much you can tune an AI model to perform optimally. After a point, you need to be optimizing systems to run these models in the most efficient way possible. That's either in terms of cost, the performance of the programs running on the device, or latency, which is the time the system takes to respond to a command. Now, when running an AI model, you can't have all three variables completely optimized. High performance and low latency would lead to an expensive model. One can't expect a model built on cost constraints to perform optimally either. Now, in the context of Microsoft, its chips are optimized for cost at the loss of a little performance.

Its CPU, the Cobalt 100, is built on ARM architecture, which puts it among the likes of Intel and AMD, who also manufacture arm-based chips. Its AI accelerator, Maya, is built on a custom architecture that's an alternative to Nvidia's closed-source architectures. You see, computing workloads on Azure are done through virtual machines. That's basically a digitized version of a physical computer. Using a CPU virtual machine enables Microsoft to control the programming and developing environment that a developer works in. Now, on the other hand, Nvidia deploys an AI model on a GPU virtual machine,

which means Nvidia takes control of the environment and consequently a larger cut of profits. It all chalks down to Nvidia's monopoly of the GPU market, leading to prices being extremely high. So if Microsoft needs to keep its pricing competitive, it has to cut down on its margins accordingly. They can reached out to Microsoft and Nvidia for the story. While Microsoft's spokesperson wasn't available for comment, Nvidia's declined to participate. Stay tuned for more. We all know by now that Nvidia has a complete stranglehold on the world's GPU supply.

So while tech companies may harbor some resentment towards the company, it's in everyone's best interest to be in Nvidia's good book. especially now, because it seems like the stranglehold is only going to get tighter. You see, NVIDIA is planning to announce a launch of its serverless API or application programming interface. It allows another application to access a program through its own interface. So, developers will be able to use output provided by it in its own application. It's called NVIDIA Cloud Functions and the company is dying to announce its launch at its GPU technology conference in San Jose in March.

Basically, NVDA's GPU users will no longer have to go through a third party's middle layer of software to access cloud functions. So there'll be no need for something like, say, Azure's development environment. Now, all of this does not mean that Microsoft will completely abandon Nvidia's tech anytime soon. In fact, the tech giant was reportedly the largest customer of Nvidia's hopper chips. People say this could be the company's way to get. gain more market share.

And the numbers point at exactly that. AWS currently dominates the Indian cloud space. It has nearly 31% market share. But Azure has steadily inched forward off late, moving up to 25% market share as of FY24 from nearly 21% the previous financial year. If its bet on its own chips pays off, it could add a significant percentage of revenue to its cloud business.

But shaking off Envidia's dominance and gaining market share is still a medium to long-term play. It's one that will require a fair bit of patience and talent. With all of that at play, India becomes even more important. Its semiconductor ecosystem is an ideal proving ground for Microsoft's chip design endeavors. The bottom line is, this won't be the last time Microsoft comes looking for talented young silicon hardware engineers at India's engineering colleges. Daybreak is produced from the newsroom of the Ken,

India's first subscriber-focused business news platform. What you're listening to is just a small sample of our subscriber-only offerings. A full subscription unlocks daily long-form feature stories, newsletters and podcast extras. Head to the Ken.com and click on the red subscribe button on the top of the website. Today's episode was hosted by Rahil Filippo's and edited by Rajiv Sien.