The Decibel - Canada’s uncertain AI future

Episode Date: April 3, 2024

Since 2017, Canada has worked to attract the best artificial intelligence research talent in the world. But it hasn’t been investing in the infrastructure that those researchers need to push the limits of AI and the questions it can potentially solve. There are now calls for the government to invest more into building out more powerful supercomputers.Report on Business journalist Joe Castaldo explains why the issue of growing computing power isn’t as simple as buying more high-end computer chips, the potential problems of investing heavily into this tech, and what Canada can do to prevent a brain drain of AI talent.Questions? Comments? Ideas? Email us at [e-mail]

All right. So, Joe, you've been covering AI for some time now. I imagine you're quite familiar with ChatGPT. We're going to start by just dropping a little question in here and see what it says. So I'm just going to pull this up. So let's be silly. Write me a IQ about springtime. So it's thinking. So Joe, what's happening here? I've just typed this in. What's actually going on? So there's a lot that goes into

providing you with an answer. Joe Costaldo is a journalist with the Globe's Report on Business, and he's been covering artificial intelligence. Probably one component that people might overlook or not think about very much is, you know, your question is being pinged to a data center somewhere in the world

where some pretty powerful, sophisticated computer chips are powering the AI model to do a bunch of calculations to write that for you. It's these very powerful and very expensive computer chips that allowed ChatGPT to generate this little haiku. Blossoms gently sway. Birds sing in the morning light. Springtime's vibrant play. Like we think about AI and digital services as being kind of ephemeral, like, oh, everything's in the cloud, just this data zipping through the air.

But it's really tangible. It's, you know, it's metal and plastic in these computer chips that fill very large data centers around the world. Joe recently had the chance to visit one of these centers in a building north of Toronto. It's just a room. It's not much to look at. It's a room with fluorescent lighting filled with these tall, black obelisks

from 2001 or whatever. But they're filled with computer chips like CPUs and GPUs. And it is very noisy in there because there are a lot of cooling fans just constantly spinning around and it creates this unrelenting whoosh. You know, you have to yell to be heard in there. Hearing protection is recommended. They're also generating a lot of heat, these chips. Like if you open up the back of one of these server racks, it's like, you know, it is like opening up a hot oven. As more and more people and companies use generative AI, we need more and more of these supercomputers. Without them, it's going to be hard to develop better AI. So today,

Joe is going to explain why this is a particular challenge for Canada. I'm Maina Karaman-Wilms, and this is The Decibel from The Globe and Mail. Joe, thank you for being here. No problem. Happy to be here. Well, let's get into this, Joe. So when we talk about the computers behind AI, I guess, how are these different from the computers we interact with every day in our lives, like our phones and our laptops? What's the difference? They're much bigger and much more expensive and much more powerful. You know, most people are probably familiar with CPUs, you know, which are found in your

laptop. And, you know, those chips are good for like general purpose tasks, like, you know, browsing the web or going on Instagram. With artificial intelligence in particular, the chip that everybody wants, they want GPUs or graphics processing units. This is sometimes used in gaming, right? This is where I've heard this. Yes. So they weren't originally for artificial intelligence. They were used in gaming and video processing, that kind of thing. They're good at taking a big complex task, breaking it down into smaller tasks, and doing those smaller tasks at the same time. Whereas a CPU does things one after another.

So GPUs are just more efficient in that way. Okay, okay. So GPUs as opposed to CPUs, and not to throw too many terms in there, but they're kind of measured using this concept called FLOPs. Is that kind of the amount of computing it can do then? FLOPs, yes. So that's an acronym that stands for floating point operations per second. So a floating point operation,

the easiest way to think about it is it's just a calculation performed by a computer, like an arithmetic calculation. Like your smartphone, for example, a top of the line smartphone can do maybe a trillion flops. So a trillion calculations per second. When you start talking about supercomputers, you're getting into what are called petaflops, so a trillion calculations per second. When you start talking about supercomputers, you're getting into what are called petaflops, which is a thousand trillion flops. And the absolute, you know, most powerful supercomputers in the world can do a million trillion flops.

And if you write that out, it's 10 to the power of 18. So a one with 18 zeros. You know, these numbers kind of start to lose meaning because they're so big, but the point is they are big. Yeah. And, you know, a supercomputer can consist of thousands of GPUs. Okay, so these are like orders of magnitudes, more powerful, basically.

I guess this may seem like a simple question, Joe, but why does AI need computers that are so powerful? Very broadly, you know, an AI model makes predictions. Like with a chatbot, it's predicting the next word that can follow the preceding words, or an image generation, like predicting a pixel. And so AI models do this by looking at lots and lots of data and finding connections and patterns in that data in order to be able to make these predictions. And that process requires computing power to go through that data and make these connections.

These days, with large language models that power chatbots, there's just a lot of data that go into training or building these models, right? They're trained on the whole of the internet, everyone likes to say. So that's a lot of data. And to do that relatively efficiently, you need a lot of processing power. Yeah, like instantaneously, like we saw, right? So this is combing through a lot of data right. Yeah, like instantaneously, like we saw, right? So this is combing through a lot of data right in the moment there. It sounds like these are pretty incredible machines, Joe.

I guess I wonder, what are the drawbacks of operating these incredibly powerful computers? Well, they are expensive to build and to maintain. And there are a lot of environmental considerations as well. They're running all the time, essentially. So they consume a lot of electricity. And there have been some studies about that, like training a large language model, for example, consumes enough electricity to power 30 homes in a year. And depending on where all this equipment is located and where the power is coming from, there are environmental considerations and emissions associated with it. And there are worries about water consumption and withdrawal as well, because a lot of data centers use water for cooling. And there have been instances where local communities are opposed to data centers in their area because of the impact on water.

Yeah, that sounds like an important consideration for sure. And with tech, we know it moves really fast. How frequently, I guess, does that kind of tech need to be updated? Well, the kind of rule of thumb that people say is every two to three years, these systems need to be upgraded, which again is expensive and arguably wasteful. But there are companies now that are working on solutions to run these supercomputers more efficiently and to get more juice out of older equipment so that you don't necessarily have to refresh every two or three years. And when we say refresh, we mean just like kind of throw stuff out and buy new stuff?

New equipment, the latest and greatest. How many of these supercomputers are actually out there, Joe? I guess it depends on how you define supercomputer, which is a very loose term. But there is an organization called the Top 500 List that for at least three decades has been tracking the most powerful computers in the world. You know, not surprisingly, the most powerful computers in the world. You know, not surprisingly, the most powerful systems tend to be located in the U.S. Japan has the second most powerful supercomputer.

Finland has one. There's some in Europe as well. What about Canada? The most powerful computer in Canada is located in Quebec, and it ranks number 134 on this list. So that's throughout the entire world, basically. Yes, as of late 2023. For context, like when the system came online in 2021, it was ranked much higher. It was number 83. So that gives you a sense of how quickly things are moving elsewhere. Not so much in Canada, though. Yeah. And these computers that we much in Canada though. Yeah.

And these computers that we have in Canada, who actually owns them? There's a federally funded nonprofit organization called the Digital Research Alliance of Canada. It grew out of another organization called Compute Canada. There is sort of public compute infrastructure here that is available for academia, so university and

college researchers overseen by the Digital Research Alliance. And there's five host sites in Canada. So there's some in BC, Ontario, and Quebec. And when you use the term compute, Joe, that's essentially AI lingo for computing power, like the power of those GPUs, those graphic processing units. I guess what I'm trying to understand about the computers in Canada, Joe, you said these are government funded. So are they also government owned? They are government funded. I think they tend to be owned by universities. So the Niagara cluster, which I visited, that's technically owned by the University of Toronto.

But any university or college researcher can theoretically access it. Okay. And so the purpose then is for these to be used for research primarily? Research in any domain. So not just AI, but anything like chemistry, biology, climate science, anything that requires a number crunching, essentially, any sort of academic study, they can be used for that. And it sounds like you said they're government funded. That seems like a lot of money for the government to spend. Are there companies, I guess, helping to pay for these as well? This is all publicly funded infrastructure. On the private side of things, companies can

build their own. They can buy GPUs and build their own data centers. That's expensive to do. It's not something that startups in particular have money to do. So if they need to build an AI model and then run an AI application, typically they have to buy access through a commercial cloud provider like Google Cloud or Amazon or Microsoft. And that's becoming harder and harder to do because everybody wants to do AI. And the other problem is even getting access. Like even if you have money, that's not enough. A lot of companies rely on established relationships they have with cloud providers in order to

get access. Otherwise, you could get put on a wait list. You have to pony up a lot of money up front. And that has implications as well, because companies, they're spending more money to do this. If access is a problem, maybe they're not moving as quickly as they could be otherwise, because they're waiting for access. Or maybe they're just scaling down their ambitions. And you mentioned, of course, this is not cheap to do. So how expensive are we talking? It's tough to put a number on that because it depends on the application. But for some companies, the cost of compute has been rapidly rising. And in some cases, they're spending more on compute than

they are on talent, than they are paying salaries. So if a company wants to build out its own infrastructure and buy their own GPUs, for example, like a single GPU is now between $30,000 and $40,000. And you need a lot of them for some tasks. And it's not just the cost of the equipment. You need land to build a data center you need people to work in the data center so it gets to be a big expense if you want to build out your own infrastructure which is why a lot of companies rely on the big cloud providers to do it for them we'll be back after this message. Okay, so there's kind of the bigger companies that can do this or governments. So in Canada, it looks like the federal government is really behind the funding

here. Joe, how active has the Canadian government been about building out an AI industry in general, really, in this country? So compared to other countries recently, you could argue they have not done much. Back in 2018, the federal government announced some $375 million to help establish the Digital Research Alliance and upgrade the systems. And then in 2021, the government announced another $40 million for Canada's AI research institutes specifically to upgrade their compute capacity. The thing is, this money was announced before the era of generative AI, which really demands a lot of compute.

So even with this money, people say it's not really enough. It's not really where we need to be. And this money is only rolling out now. So Canada is moving quite slowly in this area. And this money would double the capacity on the public side of things, like that public computing infrastructure for academia. Even then, that would still be like 10% of the capacity of like one of the largest supercomputers in the world. So what effect is this having on Canadian talent? I know we're talking about academia, Canadian universities and colleges that I guess are relying on this. You know, if we're not keeping up, what effect is that having on their research?

There are a few things that people are worried about. One is that talent will leave Canada if they can't affordably access compute to do their research, particularly in the AI field. They might relocate to another country where they can more easily do that. And Canada's really put an emphasis on trying to bring AI talent here, right? So this would be a problem for us if people are leaving. Yes. Canada announced a whole strategy back in 2017 to build up the AI research ecosystem here. And it's a real point of pride for the government. The overlooked component of that arguably has been compute. There was not a lot of funding relatively to build up that compute infrastructure to enable researchers to do their work.

So as you're saying, the overlooked component. So it sounds like Canada really focused on bringing AI researchers here, getting the talent here, but then didn't really plan on investing in the money that it would take to kind of upkeep this technology that they would need. So that's kind of the situation we're in now then. Yes, they invested some. So the three AI research institutes, that's Vector in Toronto, Mila in Montreal, and Amy in Edmonton, they do have their own GPUs, just not very many. And

they rely on that wider public computing infrastructure as well. What's changed in the past few years is just the compute requirements have increased astronomically because of generative AI and the amount of data that goes into these models. So Canada's AI strategy was planned before generative AI really took off. So a charitable explanation is the government didn't anticipate compute needs growing as quickly as they have. What has the government said about this, Joe, about building out more computing power here? Publicly, the industry minister won't say much beyond, you know, we recognize the challenges and the needs of the research community and businesses to keep pace, and they kind of leave it at that. There have been a couple of announcements recently.

Canada and the UK, for example, announced that they would work together on exploring opportunities to share, compute. Canada signed a letter of intent or memorandum of understanding with NVIDIA, which is the world's leading maker of GPUs, again, to explore opportunities to build capacity here. But other than that, not much has happened. Whereas other countries are really moving ahead. So the UK, for example, in just the last year has announced close to $3 billion to expand access there, including building three supercomputers, one of which will be or is supposed to be among the most powerful in the world. Okay, so I guess what I'm wondering then is if other countries have put more money into

this and there's maybe the opportunity for researchers here to use sites in other places, why bother trying to build up this infrastructure now in Canada, Joe? I guess I'm trying to figure out what is the case really for building more supercomputers in Canada? So there are a few things that people are worried about, both in terms of like the academia and AI researchers, and in the private sector as well, companies that want to build and deploy AI. You know, one is that people will simply leave Canada if they can't affordably access compute. Like, you know, researchers or young graduates, right? Like they might decide, you know, I'm going to move to the U.S. because there's just so much more compute there that can be accessed.

I can do, you know, really interesting groundbreaking work if I can access more GPUs. The second is the kind of work that researchers can do is limited if they don't have compute. Like I heard from the Vector Institute in Toronto that there are researchers who want to study and do work with like very big AI models, but the Vector Institute on its own simply can't support that. They do not have the compute capacity. So potentially important research does not get completed here. In the private sector with startups in particular,

they too might decide to move or start a business somewhere else, again, where they can access compute and build AI and deploy it and do interesting things. So we risk losing talent, I suppose, in both areas. And other countries that do have more compute, maybe they're able to build more profitable, more innovative companies there and capture what many see as the benefits of AI, whereas Canada gets left behind. Let's look at the other side as well, though, Joe. What are some of the risks in making a big investment in building out more supercomputers in Canada? Well, you won't hear much opposition to this idea from companies and people in the AI world. It's like free money.

Who's going to enjoy the profits from whatever they do. Yeah, because this would be government money. So kind of tax generated money, essentially, the government spending on AI that could be benefiting private companies. Yeah, public money for potentially private gain. The reality is, like, it's generative AI in particular that is driving demand for compute. And it's still a new technology. It's yet to really prove its worth, I would say, beyond, you know, a couple of smaller examples. So it's uncertain. So, you know, maybe it helps companies grow and hire more people, or maybe it leads to a bunch of job losses. Like, we don't know. Yeah, that uncertainty. I mean, when you mentioned people are looking for a billion dollars, right? If that's a billion dollars of government money going to that, that's a billion dollars not going to healthcare or education or other things, right? So there's got to be that kind of trade-off to think about too. Yeah, that's true. And so the economic case for generative AI, it's still being proven out. There's a lot of concern about low productivity in Canada. The Bank of Canada was warning about

that last week, that this is an emergency. And a lot of people think generative AI can boost productivity. There was a study by the Conference Board of Canada earlier this year that found that if every business invests in generative AI, it could boost productivity by 2%. And one economist I spoke to said, that's not insignificant, but it's also not game-changing. So there are questions to consider, of course, before making an investment like this and sort of really proving out why it's worth it. So, Joe, I imagine people are looking to the upcoming federal budget in mid-April to see if there are any dollars allocated to computing power for AI. But even if there are, of course, it sounds like that would take some time to develop. What could the government do in the short term really to work on this?

There are a few options that people have been discussing. So one is that the government just buys capacity through Google Cloud or Amazon or Microsoft or whoever. So they buy access and they develop a way to allot that to companies so that they can use it. Another thing is to just buy GPUs directly from a company like NVIDIA to ensure that Canada has access to these chips. And some countries are looking at this as a sovereignty issue, like they want compute in their own country because it's that important. So a longer term option is to invest and build more compute capacity here and ensure that there's some capacity for public

sector researchers and businesses could buy access below commercial rates. And that money would be used to maintain the system, pay for upgrades, pay for the people who work there. It's sort of like, you know, a publicly funded toll road. It's not a super complicated problem in a way. It's not like, you know, the housing crisis or the opioid crisis where there's so many different factors and levers to pull. There is a relatively straightforward solution to this problem. Joe, thank you so much for taking the time to be here today. Thank you. That's it for today.

I'm Maina Karaman-Wilms. Our intern is Manjot Singh. Our producers are Madeline White, Cheryl Sutherland, and Rachel Levy-McLaughlin. David Crosby edits the show, Adrienne Chung is our senior producer, and Angela Pachenza is our executive editor. Thanks so much for listening, and I'll talk to you tomorrow.