In The Arena by TechArena - Getting Ahead of Generative AI with the CTO Advisor Keith Townsend

Episode Date: June 26, 2023

TechArena host Allyson Klein chats with CTO Advisor Keith Townsend about how enterprises should ready themselves for generative AI integration into their business opportunity.

Welcome to the Tech Arena, featuring authentic discussions between tech's leading innovators and our host, Alison Klein. Now, let's step into the arena. Welcome to the Tech Arena. My name is Alison Klein, and today I'm so delighted to have Keith Townsend, the CTO Advisor in the studio with us. Welcome, Keith. Thank you, Allison. I'm going to get in a dig. You're like 40 plus episodes in, and I'm just appearing on the podcast, so I'm a little offended. Well, I've been saving you for, you know, the second round so that we could take it to a higher level, I think.

And we're bringing a topic that's going to take it to a higher level, too. Congratulations on some excellent content. You've been a machine turning these out. Thanks so much. I hope you've enjoyed them. I've been enjoying reading your blogs. And one of the things that you wrote about inspired this podcast. You've been writing about generative AI and chat GPT quite a bit, as have a lot of people.

But I think your viewpoint is very interesting, which is why I wanted to have you on the show. How do you look at this from a CTO advisor perspective as a disruptive force and enterprise? We'll start there. Yeah, so there's been stops and starts around these types of business-facing capabilities. We look back a couple of years ago, there was the robots movement around these like UI path and some amazing output from that, but that still took a little bit of technical capability.

Generative AI is one of those things I can give my 14-year-old, my 15-year-old. And they come out with some pretty creative stuff. I'm not just talking about report on grapes of wrath and I don't want to have to read it. I'm talking about, give me perspectives of Grapes of Wrath from someone who lived in the 1930s versus someone who lived in 2020 or across three time periods. When you give that type of utility to creatives and personnel, you end up creating production that as a CTO, you just can't predict. People are going to use this tool in an amazing and very dangerous ways. When you look at the enterprise applications, I think everyone knows that this is going to change industries. Where do you see the changes coming first?

And how fast do you think this is going to drive not just the discussions of, wow, this is what this capability can do, but real change to businesses and real change to how industries function? So let's walk this back from some typical tough problems we have in enterprise IT. I want to know if Larry has access to sensitive data he should not have access to. Good question. And the drumbeat and where we're headed is that I can access of generative AI across my systems. The algorithms and the connected tissue and the SRE type capability is almost there. So I can ask that plain text question of does Larry have access to sensitive data that he should not have access to? And the system could flag rules and firewalls or the ACLs and databases that may indicate that Larry has access to sensitive data that he should not have access to. And I can give it to my smarter people to go and analyze those controls

and see if this is a false negative or if it's true that Larry has access to data he shouldn't have access to. One of the things that I was just talking to Bob Rogers, who is a former colleague of mine at Intel, and one thing that he preaches is having a great data strategy, knowing where your data pipelines are so that you can implement things like AI. From your experience in working with IT organizations,

what do you think is the reality in 2023 about an average enterprise having a great data strategy, knowing where all of their data is? Based on what you said about Larry and having access to data that he's not supposed to, my guess is it's not very good. No, it's not very good. Let's think about the complexity of how other end-user driven technologies have changed

the landscape of enterprise IT. When I give a data scientist the option of importing all of his data set into AWS so he can put 3,000 CPU cores against a data science problem, how do I know that data scientist has moved that data to somewhere else or that he's deleted that data or that he's followed the corporate policy? This problem has been exasperated as the cost of enterprise storage has continued to reduce these cries to folks like, you know, we have friends at CloudFlare and they have this unlimited egress for object storage. And now this type of data work can be done across compute and

across environments without egress fees. So we've asked for these changes in structure of how we're charged for data and we're living through the consequences of the cloud providers giving us what we want. So the more we're giving users and systems what they want, the more we're struggling with the after effect of that Jaban paradox. Now, you bring up a really important point, which is based on where your data is might connote where your data strategy is. And that brings me into, you know, how do you build the right infrastructure strategy to take advantage of things like generative AI?

And we have so much changing on the infrastructure side. You know, we've got new technologies coming into the market. We've got the Excel potentially disrupting how data center infrastructure is built. We have chiplets. We have organizations struggling with multi-cloud. How do you see the future of on-prem data centers? Does that impact a multi-cloud strategy with all of this technology innovation going on? So let's, you know, let's extract the level above generative AI. IBM likes to talk about foundational AI or foundational models. So language is not always the best way to describe a problem. If you're talking about a three-dimensional problem or a problem that isn't using language,

scientific community, if we're talking about DNA sequence, language doesn't help us with DNA sequence. So if I'm training a model with huge data sets, I think the last time I looked, the average DNA sequence was about one terabyte. So if you're looking at a population of a million people, a million times one terabyte, you need to analyze all of this data. Or if we're talking about language or some other model, we need data centers and we need architectures that can be flexible. We're not talking about new ideas. The basic principle is that we need to get as much in-memory processing so that multiple CPUs or GPUs can take advantage of that memory.

Memory is the fastest IO within any data center. The bottleneck is the latency between that memory that's hosted in a node three racks down or across the data center to the CPU that wants to access that or the GPU that wants to access that. So technologies such as CXL, CXL switching promise to eliminate that bottleneck. And now what happens when we can put, I'm going to say a crazy number, a petabyte of memory across 3,000 GPUs. Yeah, that really opens the door, doesn't it? Yeah. Now, my real question is, some of these new architectures require some advanced skills

in terms of how you implement them. Do you think that the average enterprise is going to be able to deploy decoupled racks of memory and compute? Or is this another accelerator to taking on the public cloud? So as far as I know, you know, AWS is secretive about what they do currently. They kind of let us know what they're doing, what they did three or four years ago. So accelerator cards, et cetera, the things that the enterprise is just now looking at, AWS did three or four years ago.

So I'm pretty sure companies like AWS, Google, Azure, the big cloud providers, they're doing some of this stuff now. They're taking advantage of some of these architectures now just based on, you know, cursory what they're doing with VMware, vSphere in the cloud. You know, the driver for the storage in a host for a VMware vSphere host living in AWS, that driver is a EBS driver. EBS being the protocol that Amazon uses to attach storage to physical hosts. Not the typical NVMe driver, but an EBS driver. So Amazon is already doing some of this to some level in their data set? To answer the first part of the question,

the cloud providers are always going to be ahead of the enterprise and able to take advantage of these technologies way before the enterprise does. But we will see this skill set trickle down to the enterprise because smart people still want to work for specific companies in specific industries. A driver for someone discovering a cancer for drug, a drug cure for cancer, aren't the same driver for somebody who wants to provide generalized compute across several different industries. Now, when we look at this, the real question is, who's going to win in the industry? You know, there's so much disruption. There are new technologies coming out. There's, you know, the discussion of an open field for silicon. There's, you know, open fields for different types of infrastructure solutions. And then when we move up the stack and

get into AI, you know, a huge battle for dominance in terms of who is going to win the AI era. Who are you betting on in terms of doing interesting things? And I'll make it easier on you. You know, heading into second half of 2023 and 2024, who do you see making major moves when it comes to silicon, when it comes to infrastructure solutions, and when it comes to software? So, NVIDIA rules the rules when it comes to just pure silicon-based power around AI. They have the developer ecosystem, they have the platform, and they are leading in the silicon race. I think from a generalized compute perspective,

Apple might be an understated underdog. You look at their latest announcements around the Mac Pro. This is something that I've tried to pitch to the Intels and the NVIDIAs of the world. NVIDIA, I think, answered the bill with their supercomputer model of a data center. But Apple will sell you a system now with 192 gigabytes of RAM and 76 GPU cores. That's crazy. A generalized Linux type system with that type of capability. I don't think we've scratched the surface

of what you can do with the M.2 Ultra chip around AI and workloads. And you add to that a PCI-4 bus. I wish it was a PCI-5. Okay. And the ability to put extremely fast SSDs on the same bus, et cetera, et cetera. If we go back and paint the picture of the data center wide computer and cram that into a single node, that's only $13,000, $15,000 fully loaded. That's a really interesting piece of equipment. Yeah.

Yeah. No kidding. I am really excited to see what the ecosystem does with different types of configurations, because, you know, that's my sweet spot. I love to see what type of hardware configurations come out. I know that's probably not the most exciting area for most people. And I'm looking forward to seeing what the OEMs, are there any disruptor OEMs that are coming out with different infrastructure with this? I think we've seen some early players like MemVerge come out with some interesting boxes for memory pools, and I'm sure we'll see more. What do you think about the software front in terms of the large cloud players for dominance in AI? Yeah, so I think Google and Microsoft

are a little bit ahead of AWS. AWS is kind of pulling up the rear with their options. Google bet early and big on TPUs and software tools around enabling AI. They've been talking about AI for at least five or six years now. And Microsoft made the big investment in open AI and chat GTP and this paying off with ease of use and being able to make option of these technologies easier.

Developers are still front and center. And I think one of the things that we can conflate is developer and data scientists. Most developers aren't data scientists. So they don't know how to take advantage of AI. They don't know how to fully take a huggy face algorithm and apply it to their algorithm. There needs to be this middleware. And Microsoft has done a really great job of raising their hand and saying,

we will be the middleware for your AI in the cloud. And Google has been raising their hand and saying, we'll be your source to create and run models at. And then AWS has said, you know what, we'll figure out, we'll help you figure out how best to run AI against your S3-based data lakes. One final question for you. We went at the beginning of this interview into the fact that enterprises are not ready in terms of data policies to fully adopt AI. We've talked about the opportunity statement and how fast the cloud players are going in this space. What is your advice in 2023 for enterprise leaders to get their acts together with their data and make sure that they are the disrupting force in their industries when it comes to AI adoption.

So step one, create a data policy. You'd be surprised how many organizations have done basics of what are our data policies? What is the governance model around data and how we control data? Step two, do a gap analysis of where your policy is and where your ability to implement and audit that policy. Basic data hygiene. If you're already there, and I suspect the majority of people listening to this aren't there, and that'll be enough for them to your company and what guardrails do you want to put around it guardrails are super important because if you look back at your data governance policy that policy may say that you cannot create redlining effects for loaning money to or mortgages to minorities. When you put unlimited data analysis capabilities

in the hands of a bank branch manager or a private lender, they may go and file, not deliberately, but they may go and file that policy. How do you control your governance around that? I guarantee you those two things are hygiene that most companies have not done yet and will keep you busy for the first half of 2023 and going into the second half of 2024. It is the major problem that I see in cap. I love that. Keith, it's always a pleasure to talk to you. Where can folks engage with you if they want to learn more about your thoughts here and potentially engage with you in terms of their own journeys? Yeah, you can find us on the web,

thectoadvisor.com. All kinds of blogs, videos, sponsored content, all of that good stuff. And I'm on social media, most platforms at CTO Advisor. So Twitter, Blue Sky, etc. And DMs are generally open. Fantastic. Thanks so much for being with us today. It's been a real pleasure. Thanks for having me. Thanks for joining the Tech Arena. Subscribe and engage at our website, thetecharena.net. All content is copyright by the Tech Arena.