Utilizing Tech - Season 7: AI Data Infrastructure Presented by Solidigm - 2x14: Using ML to Optimize ML with Luis Ceze of OctoML

Episode Date: April 6, 2021

Training and optimizing a machine learning model takes a lot of compute resources, but what if we used ML to optimize ML? Luis Ceze created Apache Tensor Virtual Machine (TVM) to optimize ML models and has now founded a company, OctoML, to leverage this technology. Fundamentally, machine learning relies on linear algebra, but how should we pick the fastest approach for each model? Today this is done with human intuition, but TVM builds machine learning models to predict the best approaches to try. It also creates an executable so the model can run best on various target hardware platforms. It can also help select the right target platform for a given model. Three Questions How long will it take for a conversational AI to pass the Turing test and fool an average person? Is machine learning a product or a feature? Can you think of any fields that have not yet been touched by AI? Guests and Hosts Luis Ceze, CEO of OctoML. Find Luis on Twitter at @LuisCeze. Chris Grundemann, Gigaom Analyst and Managing Director at Grundemann Technology Solutions. Connect with Chris on ChrisGrundemann.com on Twitter at @ChrisGrundemann. Stephen Foskett, Publisher of Gestalt IT and Organizer of Tech Field Day. Find Stephen’s writing at GestaltIT.com and on Twitter at @SFoskett. Date: 4/6/2021 Tags: @SFoskett, @ChrisGrundemann, @LuisCeze, @OctoML

Welcome to Utilizing AI, the podcast about enterprise applications for artificial intelligence, machine learning, and deep learning. Each episode brings in experts in enterprise infrastructure to discuss applications of AI in today's data center. Today, we're discussing the novel concept of using machine learning to train machine learning. First, let's meet our guest. Hello, my name is Luis Tazi. I'm co-founder and CEO of OctoML and professor of computer science and engineering at the University of Washington. I've done a bunch of work at the intersection of hardware and software for machine learning. And you can find me at Twitter at Luis Tazi.

And I'm your co-host today, Chris Grundemann. I am a consultant, content creator, coach, and mentor. You can learn more on chrisgrundemann.com. And I am Stephen Foskett, publisher of Gestalt IT, organizer of AI Field Day, and a host here of Utilizing AI. You can find me on Twitter at S. Foskett. So Luis, when we were speaking previously about your work, both as a researcher and also with OctoML, the topic came up of the challenges of machine learning training, not just in terms of the tasks associated, but also in terms of sort of that apocryphal story about how much power and global energy is being used and all that. Your insight was, well, hey, how about we use ML to train ML?

Do you want to tell us a little bit about that background? Yeah, absolutely. So, as you know, with machine learning getting more and more popular and getting all models with all sorts of useful things like computer vision, language understanding, content generation, and so on. One thing that's important to realize is that the underlying compute power that they use is significant, right? So when you are doing computer vision or, say, applying a GPT-3-like model, you're doing billions and billions of calculations to actually provide the answer. And that translates to a lot of compute resources that is used and needs to be optimized to

be viable. Typically, a machine learning model without a significant optimization is many times too slow to be deployed or use too much compute resources in the cloud, which translates to high costs and also translates to energy usage and so on. And the concept that we explore started as a research project and I was part of the Apache TVM technologies to use machine learning to optimize the resource usage of machine learning models

to make them use a lot less resources. Think of it two, three all the way to 30X better performance and lower power consumption. So. 30X better performance and lower power consumption. Wow. 30X is obviously a huge improvement there. Is TVM an acronym? Yes, it stands for Tensor Virtual Machine. So as you know, we use the virtual machines in all sorts of ways in IT infrastructure, right? So think of the Tensor Virtual Machine, which is now Apache TVM, part of the Apache Software Foundation. It is essentially a thin layer on top of all sorts of different hardware machine learning models run.

Think of it, all the CPUs, GPUs, and accelerators in the cloud or all of the systems on the chip on edge devices. They all have their different ways in which they execute code and different ways in which they provide better, more efficient computation. And figuring out how to utilize and make the best use of that hardware today is done by hand by and large. Like you use hardware vendor specific software stacks like NVIDIAs or Intel's or et cetera.

And so data scientists have to be paired with software engineers to go and optimize their models, get them ready for deployment. And what we did with Apache, with the TensorFlow virtual machine work is to turn that into an optimization problem that uses machine learning. So we essentially find the best way in which your machine learning model would run and the target hardware that you want. So it really is analogous to a compiler for regular computer programming code, right? So, I mean, it's taking, you know, the code and basically optimizing it and targeting it to a specific machine and the capabilities of that machine in order to make it run best. That's exactly right, Steven. So, in fact, if you go to our, you know, open source project website, you know, tvm.ai or

talk to ML, you're going to see the word compiler everywhere. So, TVM fundamentally takes in a model as input written by the data scientists in TensorFlow or PyTorch, whatever it is, and then translates into an intermediate internal representation and then applies a lot of compiler techniques to produce an executable out, right? So typically the state of practice today

is to use some sort of predefined libraries and the models are largely interpreted. The way TVM does it and OctoML does it is actually compile your model into an executable that's tightly optimized for the specific hardware target. And we can get later if you want to hear more about why I use machine learning for that problem, because machine learning is unique in terms of its computation. And if you think it makes sense, I can tell you more now or later. So. Yeah, I mean, I think that makes a lot of sense. I'd love to understand. I mean, cause on one hand that almost sounds like a Myerisch-Gaddal or something, right? If you're

using machine learning to make machine learning better, there's a bootstrapping issue there, right? So I'm really interested in how that works and how you approach that. Yeah. The bootstrapping process here is starting with a naive implementation. So here's the problem that we're solving fundamentally. A lot of machine learning models, when the processor is executing, boils down to multiplying matrices and vectors and doing convolutional sorts of ways. Large bags of numbers that you do a lot of linear algebra with. So it boils down to a bunch of loops and different ways in which you can organize the data in memory, which instructions using the processor, and so on. And for every single machine learning model component, there's literally billions of ways in which you can compile that into an executable for your target platform. Okay?

The question, and they're all equivalent. They all produce the same results. The question is, one of them is 100 or 1,000 times faster than the other. How do you pick the fastest one? Okay? So you cannot try them all because it's just too compute intensive. You're going to try, if each trial takes a second and you spend, you do tens of billions, you're talking about centuries of compute time that you can't afford. So today this is done by hand, by using human intuition. And you have people that scrub the

low level math libraries to go and figure out, here's how I'm going to organize these codes to go and make it run faster. Right? So the way TVM does it, it builds machine learning models that predicts how a piece of code should execute in a specific target platform and uses that to guide this exploration of how the code should be compiled. So think of it as compilation plus search. So you're compiling a bunch of different choices. You want to pick which one is the best one, but you cannot try them all. So you need to build a way of predicting how you should optimize and comply with that piece of code that your

machine learning model. And that's what TVM does. We do that for all sorts of hardware. And at OctoML, we have a library of hardware personalities, which are these models that predict how the machine behaves to be able to guide the optimization of machine learning models to the target platform without having to rely on expensive and slow hand tuning of these codes. Does that make sense? Yeah. And actually that leads me to the next point, which is that there's also an aspect here of tuning the code to a specific target platform.

So not only are you finding, quote, the best approach, you're also figuring out the best approach for these specific target platforms. And that can be a very powerful thing, both in terms of distribution of a model, but also in terms of optimizing execution of a model in production, right? So you're gonna talk to the ultimate end

where that model will be run, and you're going to figure out, okay, what are the platforms that they're going to be using, right down to what the specific chips are, and then basically deploy the right model at the right place. That's exactly right. So some of our current customers' use cases are doing, say, large-scale computer vision over a large collection of videos in the cloud, right? So to do say content moderation or figure out what's in those videos, right? So, and that amounts to a ton of computation. In that case, we offer 10X better performance on their chosen GPU compared to the version

that was hand tuned by the engineers. So, and since these models change all the time, make it automated, right? So you can update your models and then get them optimized and redeployed. It's a significant engineering time-to-market savings and cost savings. There's one more point, too, Steven. You pointed out you can choose and get optimized on the four-year chosen platform. The other use that we find interesting, too, is that when you don't know what's the best platform to deploy, say that, for example,

I give you a large-scale language model. You're building a chatbot or you're analyzing large-scale documents. We're going to run this and you have a model that you customize for your use case, say for legal documents, okay? So you as an application developer

or user business, you're going to go and optimize that model and then you customize it so it's no longer a standard model as an application developer or use a business, you're going to go and optimize that model. And then you customize it. So it's no longer a standard model. It's right there in the wild. And now you're going to run it at scale. Okay, so, and you're going to run it in the cloud.

Which GPU should you use? Some instances are going to give you a much higher throughput per dollar than others. And this difference could be high. And right, so it could be 10X, could be 20X, more cost more cost efficient to pick a specific GPU that you might not know which one it is. If you're doing this by hand, you have to try them all and figure out which one is the one that you should run it. We can do it automatically for you. So we get your model, tune in for all harder targets and pick the one that gives you the best results. Yeah, that's really interesting too, because of course there are a lot of GPUs out there and picking the wrong one can actually

be a pretty costly mistake or could even stop you from being able to deploy a model in a certain configuration. So, you know, for in the data center, I think most people have plenty of power and plenty of cooling and they could basically pick anything. But even in that situation, they might be picking the wrong GPU. They might be specking out some kind of, you know, high-end GPU to perform, to run this particular model and this particular task when they might not actually need that kind of resources if the code was just optimized a little better. And then at the edge, of course, this is just a tremendous difference because it could be the difference between being able to run

your model on a edge device or a mobile platform or something like that and not being able to run your model on a, you know, edge device or a mobile platform or something like that and not being able to run it at all, right? Very, very well said, exactly. So the way I like to think about this is either you save costs in the cloud or you can scale more with a given fixed hardware budget or for deploying the edge, there's one thing that you can't do is just magically change the target platform, right? So if you're going to deploy this on Android phones, for example, the hardware is out there, right? So you have to make your model run in 100 different hardware configurations, and you can't just magically replace the phone. So you have to make your model fit those hardware resources. That's absolutely right.

I like to say that optimization typically sounds like a nice to have, but in machine learning, it's a must-have. Without optimization, you just can't deploy it right so so we're talking about the edge and even all the way down to phones and things i mean does that mean that this works both for for training as well as inference or does it have a specific use case within kind of the evolution of a model no uh great question right now we are uh focused on inference because even though training gets a lot of attention because it's costly and you have to wait until you have your model out, and so by and large in a lifetime of a machine learning model, most of the computation is in inference because you train it once and you do billions of inferences, right?

So the bulk of the computation overall is in inference and that's where the bulk of the cost, if you're going to deploy it at scale. So this is our... and also in edge devices, you do inference most of the time. You rarely do, you might do some local training and refinement, or you can do something like federated learning, or you do local data gathering and you do local model refinements.

But, you know, we're focused on inference. That said, Apache TVM, we are actively working on supporting training job optimizations as well, where instead of, like I was describing before, tuning a machine learning model to do inference in a highly optimized way in your target hardware, we are working on taking as input your training job and optimizing it for the training hardware. Does that make sense? So then, because in all these savings of significant performance gains and lower costs can also

be applied to your training jobs. Our aim is to just cover more and more like for things that use a lot of computation and takes a lot of human time in setting it up. This one we want to automate. We're fundamentally in automation, machine learning acceleration platform. We want to automate that. So focus on inference, then covering training, and then eventually doing the full deployment. And I think that this is one reason that this is relevant in the enterprise space as well, because I think that on first blush, you might look at this and be like, oh, well, this is really a tune just for the modeling people, the researchers, the academics, you know,

we're trying to like actually, you know, put this stuff into production, but it's important to put it into production. It's important to optimize it. I mean, if I went to most end users and said, you know, you've got a choice, you can either use, you know, interpreted, you know, shell scripts or, you know, basic programs, or you can use a compiled C program, you know, what are you going to do? I think that they would be actually very interested in the answer to that question. And I think that to me, that's kind of what I'm looking at here. I'm saying that this, this really is a practical solution that helps us to make practical use to really utilize AI instead of just, you know, sort of messing around with, you know, what it could do, right? Yeah, right. And that's a beautiful analogy, Stephen, like, you know, using interpreted script versus compiled code.

That is exactly what we're playing here. We are offering the compiled code version of machine learning model deployment in your analogy, right? So, and I want to emphasize that, yeah, so most of our open source Apache TVM users are now practitioners. It's not just used by researchers. It started as a research project five years ago. But if you look at the evolution of our conferences even, so last conference that we did on Apache TVM, we had about 1,000 people.

Most of them were ML end users that are interested in getting their models deployed. And in the set of big users of TVM, you see things like Amazon, Qualcomm, Microsoft, Facebook, they're using that to compile their models and deploy them. So yeah, we want to make this technology of optimizing compiling models into efficient binaries something that is very easy for anyone who can get their hands on a machine learning model or create a machine learning model to go and use it. Yeah, that's great. That's awesome that it's getting practical use out in the wild.

I wonder, does this actually work the other way as well? Can you use this to decide between models? So I wonder if I could, I mean, because you obviously, you said you can optimize for specific hardware and even choose which hardware to use based on what would be most optimal for that model. Is there any situations where I'd have competing models and decide which one to use based on the hardware I have?

Or is that not really a problem to face? No, they do. Yeah, it's a great question. People do a lot of, they call this either network architecture search or model selection based on the hardware aware model selection or hardware aware network architecture search, which means that you're trying to search for a model that better fits the deployment hardware you have. And yeah, so we can totally help with that too

because you have a set of candidate models, right? So they all have equivalent statistical efficiency, equivalent prediction accuracy, right? So you wanna pick which one gives you the best performance and the lower resource consumption, and you can run all of them through Apache TVM or through the optimizer and pick the one for the harder target that gives you

the best performance, absolutely. So I see that as an outer loop of using your technology that you can sweep over the candidate models and pick the one that services you best. Yeah, very cool. Because I mean, definitely, again, as we're talking about the edge and phones, right, you don't really have much choice in that case, right? If you wanted to run on folks' phones, you have to deal with what's out there. And so then understanding how to optimize your model to do the inference on the phone makes a lot of sense, right? Absolutely. Remember that phone is not a single target, right? So if you think about all those smartphones out there, you're talking about hundreds of

different chipsets that have different, you know, CPU architecture, microarchitecture, different GPUs, different machine learning accelerators, and you need to work with them all. So it's rare that you have a single output optimized model that would be deployed all around. It has to have multiple versions, right? Well, especially today. I mean, if you look at this hardware, it seems like every week we're hearing about a new chip with an integrated machine learning accelerator of one sort or another. I mean, Intel, you know, just released that for the mainstream desktop CPUs. And, you know, obviously Apple and, you know, Huawei have implemented that in chips. And certainly we see, we expect that we'll see a

lot more of this stuff coming to Qualcomm. Does this bring up a challenge for you in terms of, you know, being able to target all those platforms? I mean, how do you target a new chip or a new processor platform, new accelerator that you've never seen before? Yeah, it's a great, great question. And in fact, I call that an opportunity. That's how we started, to be honest. So if you don't mind doing a turn of time, going back five years ago, six years ago, the reason we started this at that time was a research project at University of Washington was exactly that observation that you have. A variety of models that people care about and growing fast, a variety of harder targets

that we call the Cambrian explosion of harder targets. And you said it very well. It's actually accurate to say that it's roughly one a week. If you look at overall on average, you're going to get one's roughly one a week. If you look at overall on average, you're going to get one new hardware target a week in a year. So, and our key research question there was, it actually automates the process of not just compiling the model, but also onboarding new hardware.

So the same engine that I described earlier, that helps you figure out how to run a given piece of code best, like part of your model best on a hardware target, can be extended to learn about new hardware targets. Okay, so here's how this works. And I was going to get, if you bear with me for just a second, you're going to get slightly more involved. So when you design a new hardware architecture, what do you do?

You change its memory hierarchy, its caches, or you add new instructions, or you add a new accelerator. And you write, like the hardware vendors, write a thin layer of software that can compile basic pieces of code to those targets. The question is, how do you compose that into a piece of code that makes the best use of that? We use machine learning to do that. So we run a large number of experiments over that new hardware and discover how bigger and bigger pieces of code best run on that hardware target. Okay. So let me give you an example. You might have seen it was in the media in December, our results with the Apple M1.

So the Apple M1 came out, right? So it was released and in two weeks, one of our engineers only was able to actually onboard the Apple M1 chip into our platform, into Apache TVM and compile a birth model. And we had 1.5X better performance than the native library there. Because we applied this, like we plug in the hardware, we run a bunch of experiments,

we build its model of how this new hardware behaves, and we use that model to guide how to optimize existing machine learning models into this new hardware. So basically the one words to translate is automation. Like we are all about automation, automating onboarding new types of models that people would tune by hand and automating the process of onboarding new hardware targets. And it's exactly what TVM was designed for.

And it's exactly why OctoML is essentially Switzerland for that. You know, we're not tied to any specific hardware target. We work with AMD, with Qualcomm, with Arm, and we support Nvidia and Intel because we want to serve the machine learning and user community to make the most out of the hardware that they need to run their models to. Yeah, it really does remind me, basically at the start of my career,

it was when there was just this explosion of CPU architectures. And at that time, that was when there was just this explosion of CPU architectures. And at that time, that was really a point when compilers really were a major focus of computer science research, because suddenly you had, you know, PA risk and spark and MIPS and, you know, all of these other, you know, platforms out there appearing and, and you had to write for them. And nowadays, I think that people are kind of lulled into this fake sense of, well, everything is x86 and ARM. But when it comes to machine learning, that's really not true because you are perhaps running on x86 or ARM instructions,

but for the most part, you're running them on these accelerators. And these accelerators can be dramatically different. You mentioned the M1, you know, Apple's, you know, machine learning environment is so much different from Huawei's and that's just two ARM CPUs. And then you get into the, like you said, the idiosyncrasies of those things. You've got the M1, which is different from like the A14, you know, platform, right? And I mean, do you rely on the hardware vendors to partner with you, or are you just going to go out there and go to the store and pick one up and see what works? That's a fantastic question. Wow. So, and in fact, I do have a blog post, it was probably six or eight months ago that makes exactly that analogy that we have major lack of portability

today that we take for granted. It's almost like the PC era in the 80s, you know, the early 80s, where, you know, you had to, you had lock-in and hardware and stuff, it's hard to move stuff around. Yeah, so let me just start by saying that you're absolutely right. So today, when you are thinking about a model that you're actually going to put in production, you go and work with the hardware vendor-specific software stacks, sometimes specific to a specific model of a specific hardware vendor specific software stacks, sometimes specific to a specific model of a specific hardware vendor. Okay, so you have to go and pick the software package for the hardware target

and then tie that your model development to that. So we want to abstract that away, even using the hardware vendor technology. Like we want the machine learning end users to not have to worry about that. So how do we do that? Well, as I said before, we, the core, I'm going to repeat for the third time, using machine learning for machine learning here, like we use a discovery process. Yeah, we onboard, we assume there'll be a basic compiler

there. And yes, we do partner with some hardware vendors very closely. So if that's public, for example, we work closely with AMD and with Qualcomm in bringing their hardware into the TVM ecosystem. But once the basic harness is there, when it's bootstrapped, everything else happens by running a bunch of experiments and automatically running little snippets of code experiments on this hardware and discovering what they can do and use that to build a strategy on how to compile new models. Does that make sense?

So we do, we need some exposure to their hardware. We do not need any specialized detailed hardware architecture design that will be proprietary. In some cases, we can help hardware vendors, the ones that are our customers, to navigate some of that. But what's open source in TVM, anyone can use. And in fact, the community helps onboard new hardware too. And I should say, one of the things that makes all this machinery work is that we have, as of this week, we crossed the number of 500 contributors in Apache TVM. The people writing code support new hardware, support new models, support new frameworks.

And that keeps the future-proofness of the whole ecosystem. That's exciting. And obviously, these feedback loops are really interesting to me and i wonder i mean i know this is the utilizing ai podcast we've been talking kind of about the practical uses and the fact that folks are using this not just in research but i'm actually super curious about some of the you know continued or ongoing or future research cases i mean it feels like there's a real possibility for tvm to be used for innovation um maybe on the hardware side but definitely on the model side right i? I mean, if you're able to understand

how things can optimize for existing hardware, you can maybe build better models or come up with actually new ideas in ML. I mean, is that an area that folks are using it for? Yeah, absolutely. So in fact, if you look at all the leading AI ML conferences, many of those papers use TVM. There was one that I can highlight here from last year

from the Allen Institute of Artificial Intelligence here. I think the new language model that was very compute intensive and they used TVM to actually make the model viable. And that was part of the loop of how they developed the model and yeah, TVM is using research, instead of research, because of our conference that actually has some figures on that.

So of the folks that attended, as I mentioned before, about a third of those were researchers from research institutes and universities that are interested in using TVM. Some of them are already contributing to TVM as part of how they develop new models and make them more viable for optimization. Let me just pick up one more thing that I can't resist sharing. So why is portability important, by the way? So portability is important here. It's not just so you actually run on the hardware that's already out there. It's also because if you have automatic portability, it leads to cost savings as well. We were talking about earlier in the discussion today on how do you pick a cloud instance that gives you a better throughput per dollar,

gives you much more cost-effective execution. That relies on you having to move from different models of GPUs. That would be wildly different. It would require a whole retuning. If you could do that automatically, now you can just move around and pick the ones that pick, pick the harder targets that better serves your needs at that time without having to redo any manual engineering work, right? So portability is not just about it's cool and convenient, it just leads to, you know, a very bottom line cost saving zone. What would you like people to kind of take

away from this conversation who are more on the IT side and more thinking about deploying machine learning applications in their environments? Yeah, so it's all about utilizing your hardware very efficiently. As you know, people can't get enough of that. And we want to enable models to make the most out of their hardware. That's why we're on machine learning acceleration platform.

And TVM is an automated engine for that. Is that we want to do that without you having to hire specialized engineering to optimize your models. So the takeaway here is to make the most out of it. Do not think that machine learning doesn't need optimizations to be deployed. It does need it. It could be the difference between viable or not viable. And you want to do that in an automatic way.

Yeah. Yeah. And, and, and absolutely. And it can also result in major cost savings or, you know, not needing to invest in new infrastructure to support some of these things. So I really appreciate that. Thank you so much. As you know, at the end of our utilizing AI podcasts, one of the things that we like to do is throw a couple of curve ball questions at you generally about the state of

the AI space. And now's the time to do that. So I'll remind the audience that I did not give Luis a heads up on these questions, and we'll see how he answers them here in real time. So question number one, how long will it take for us to have a conversational, like a verbal AI that can pass the Turing test and fool the average person into thinking that they're talking to a real person? I think we're very close to that. There have been some isolated cases of that. I'll guess within five years, that's going to happen very easily. Excellent. And, you know, this one I think is going to cut to the very heart of you and your work. Do you consider machine learning to be a product or do you think that it's a feature of a product? Well, I think more and more of the functionality that makes a product be attractive is because of the machine learning in it.

I think it's more and more the product. All right, great. And then finally, thinking about all of the fields that have been touched by artificial intelligence already, can you think of any fields of human work that have not yet been touched at all by artificial intelligence? Wow. Well, there's one that I'd like to be touched more. I think life sciences have been touched, but to some extent, I think there's a lot of opportunities in life sciences, protein design, vaccine design, that I think it's just, we are on the verge of, I think has been touched in research, but hasn't been touched in broad use yet. So I do imagine a world where there's a

new pathogen, you know, sounds familiar, right? It's just going to easily spread all over the world. How do you design and put into production a new vaccine in a fully automated way? And I do think that's something that AI might have helped a little bit this time, but I think has the potential to help tremendously in navigating extremely complex design spaces, which is a lot of what life science is about. Well, thank you so much for those answers. It's always interesting to hear how some of the answers are consistent and some of them diverge in our questions. And of course, we've got more than those three.

So just to keep everybody guessing. Well, thank you so much for joining us today. Where can people connect with you and follow your thoughts on enterprise AI and other topics? Well, you can find me on Twitter at Luis Sazi. You can also go to octamil.ai if you want to learn more about our machine learning exploration platform. How about you, Chris? What have you been up to recently? Yeah, you can find out on my website, chrisundeman.com, or follow me on Twitter at Chris Grundeman. And as for me, I've been busy working on the next AI Field Day. If you'd like to join AI Field Day as a presenter or as one of the delegates who gets to join in the conversation, please go to techfielday.com, click on the AI icon, and you can learn more

there. You can also connect with me at sfosket on Twitter or sfosket at gestaltit.com, click on the AI icon, and you can learn more there. You can also connect with me at sfosket on Twitter or sfosket at gestaltit.com. So thank you everyone for listening to Utilizing AI. If you enjoyed this podcast, please do rate, subscribe, and review the show on iTunes, since that really does help on that platform. And of course, please share this episode and all the others with your friends. This podcast is brought to you by gestaltit.com, your home for IT coverage from across the enterprise. For show notes and more episodes, go to utilizing-ai.com or find us on Twitter at utilizing underscore AI. Thanks, and we'll see you next Tuesday. day.