Utilizing Tech - Season 7: AI Data Infrastructure Presented by Solidigm - 06x10: Using AI in Modern Applications with Matthew Wallace

Starting point is 00:00:00 As we consider the impact of AI on modern applications, we should consider the ways that this technology will improve products. This episode of Utilizing Tech brings Matthew Wallace, CTO of Kamawaza AI, to discuss the adoption of AI in application development with Allison Klein and myself. Welcome to Utilizing Tech, the podcast about emerging technology from Tech Field Day, part of the Futurum Group. This season of Utilizing Tech is returning to the topic of artificial intelligence,

Starting point is 00:00:29 where we'll explore the practical applications and the impact of AI across the range of enterprise IT. I'm your host, Stephen Foskett, organizer of the Tech Field Day event series, and joining me this week as my co-host once again is Allison Klein. Welcome back. Hey, Stephen. It's always a pleasure to be here. So last week, Allison, we talked to my colleague, Paul Nashawati, about the impact of AI on application development. What did you think of that? And where do you think we should go this week?

Starting point is 00:01:00 You know, we've been doing podcasts for a couple of years together, Stephen, and that talk with Paul has got me thinking in a way that very few other episodes have. I thought some of the things that we talked about were really interesting as he was looking at how are enterprises actually going to adopt AI and large language models and introduced tons of new topics that I would love to continue to unpack here. Yes. And the three of us, Paul, you and me, were part of AI Field Day back in February. And one of the companies that presented there as a guest of Intel was a company called Camoaza. And so we have decided to invite back on the podcast this week, Matthew Wallace, to talk a little bit more about implementing AI in enterprise applications.

Starting point is 00:01:52 Welcome to the show. Hey, thank you so much for having me. I'm excited to be here. So tell us a little bit about yourself. Why should we care what you have to say? Well, you know, I was an early drinker of the AI Kool-Aid. So I've been, you know, doing this now longer than ChatGPT has been around, which I think is what marks the beginning for a lot of people. But, you know, I have a really broad set of experiences in technology, right? And some of the folks who've been in IT for a long time, you know, I worked at VMware, I helped design their first cloud product and launched their service provider program. And I've worked for networking companies like Level 3, where I ran the cloud provider relationships on the technology side.

Starting point is 00:02:33 I've done multiple startups. And that kind of range and then diving into co-founding Kamiwaza, where I'm the CTO, I think gives me a lot of perspective, not just for the application level. But what does this mean in terms of operations, in terms of infrastructure? I think some of that thinking is certainly what led into the product that we're building and what it can do. But also, I think it gives me a great opportunity to talk about what this means for the future of both software and IT kind of alongside each other. When you look at that challenge and you think about the enterprise landscape, where do you think we are with enterprises really adopting AI in mass across all of their business lines? And where do you think the industry needs to go to assist in the forward progress of that adoption? Yeah. You know, obviously there's a sort of bell curve of adoption that you find people doing.

Starting point is 00:03:31 And it's the same, I think, with this technology as with any. And out on the leading edge, of course, you have like the technology companies themselves, right? Hyperscale cloud providers serving customer needs. Other businesses, especially large SaaS providers that could muster the talent to be really early adopters. And of course, we've probably all read headlines about folks like Netflix offering people $900,000 salaries to do AI work. And I'm sure that plays a part in some of these businesses being able to poach talent. But outside of that bleeding edge, what I find is that the vast majority of enterprises

Starting point is 00:04:06 are somewhere between rough POC tire kicking, and we're really excited about it, but we haven't figured out what we're going to do yet. And so that's a pretty big range. And it's pretty nascent if you think about where we think this will go. And I am a pretty firm believer that it's going to touch almost everything that we do as you got a number of years it's hard to say how many years but for sure i think it's going to be really pervasive and so i think and this is back kind of backed up by surveys but it's still early days but because this is going to be i'm very convicted a huge huge impact on the industry I'm definitely recommending that everybody really

Starting point is 00:04:45 kind of jump in with both feet because the time to learn about it is now. Yeah. I want to bring back what we talked with Paul about last week. He had a bunch of the numbers that back up what you just said, and his numbers are absolutely in line with that. Although the adoption of AI in the enterprise is expected to be very high. Right now, it's still very low. But I think that you professionally at Kamawaza and you personally, as somebody who's bringing AI to market in all these applications, I think what you're experiencing really shows what you said and what I said to Paul last week, which is basically when we next do this survey, those numbers are going to be very different.

Starting point is 00:05:31 How quickly do you think that AI and large language models are going to come to market across the spectrum of enterprise applications? I actually want to draw a corollary to what happened in cloud, because I think there's an interesting point to make here about adoption, right? If you think about cloud, a lot of that was about how you deployed and scaled an application. And, you know, it either was a new way to do it as a startup, or it was something that replaced traditional infrastructure. But at the end of the day, it was infrastructure. Now, obviously, things like serverless took you to a place where you were replacing infrastructure with an API that could do the things for you with code that infrastructure would do.

Starting point is 00:06:13 So I want to draw a contrast to generative AI, because if you think about gen AI, I think it touches so many different use cases, right? From the coding assistants to the advanced marketing to the copywriting to the truly,, to the copywriting, to the truly, and where we think this is going to go is to a place where we actually automate a whole lot of communication and thinking and processing, really to kind of put the world of information at your fingertips. I think it's really hard to kind of imagine how pervasively it's going to be.

Starting point is 00:06:43 I mean, it's hard to go back in time and think, what was it like when somebody did bookkeeping on a physical ledger, right? We have had spreadsheets for so long. And I think that the kind of pre-AI, pre-generative AI world will someday look very much like that. We will be looking back at today and going, it was like we kept paper ledgers. But I think that what we're going to see is we're going to see patterns emerge where somebody has a good paradigm for doing something really well, backed by tools that are available, whether they're open source or commercial. They'll kind of evangelize something that they've built, a new way of doing things. And then you'll see people get excited right across enterprises that go, we can do this too. They'll kind of glom onto it.

Starting point is 00:07:20 They'll put their own spin on it. They'll get it deployed and there'll be that evolution. But it'll hit many different areas, I think, at different paces. And it's hard to say both when that starts and when those different kind of things land, because it really is up to all the individuals that are kind of working on those things. But I think you're going to see a much more scattershot adoption, I think, to get to the final state, if there is a final state. I love your analogy of the spreadsheets and paper ledgers. I also think about calculators. You know, I'm old enough to have had math classes where they wouldn't allow calculators. And now it just seems somewhat, you know, archaic to do something like that. They just are part of life. One question that I have for you is actually about tools. You know, we're in early stages of adoption, but are there any tools on the market that have come out where you've been really impressed or have you seen any that seem to be gaining traction as being the leading adoption candidates amongst enterprise?

Starting point is 00:08:26 Yeah, boy. First of all, I mean, it depends on the kind of tools that you're talking about. Right. And because there's things get divided into different buckets here. Right. And there's different phases of the kind of life cycle of levering generative. I'm right. So there's things like where am I going to get a model?

Starting point is 00:08:41 How am I going to deploy it? How am I going to evaluate it? How am I going to evaluate it? How am I going to build an application that uses it? You get into things like, how am I going to fine tune it? How am I going to connect it to my data with RAG, retrieval augmented generation, or DAG, data augmented generation, you'll also hear. And I think those questions, they all have sort of different answers. I would say, you know, for tooling, like from a developer standpoint, I'd say there's a really broad adoption, a shockingly large adoption of, say, Microsoft Copilot, GitHub Copilot right now. But I think

Starting point is 00:09:17 what I'm seeing from the tools and the development side out there is unbelievable. And of course, timely, this is, you know, only a day or two old as we have this conversation, but like the Devon launch from, I think it's Cognition is the name of the startup, right? Pretty amazing. It's like a baby AGI auto GPT type engineer,Sphere. It's a kind of AI first fork of Visual Studio Code, right? These are fantastic tools. And I have no qualms in saying that I feel truly like the set of AI tools that I use day to day truly makes me about 10 times, if not more, as productive as I was before. I think I'm maybe like the perfect target market for it, but I still think it's pretty incredible. And then by not sleeping, I can do the work of 20 people. But it's really fascinating. And then if you flip over to the other side on Gen AI, though, there's this incredible evolution going on. So Meta deserves a

Starting point is 00:10:17 lot of credit here for open sourcing Lama 2 in an almost completely open way, right? They kind of carved out a few of their competitors. But there has been such an incredible evolution of model fine-tunes, model architectures, both on Lama and then also, credit to Mistral. Mistral and their mixture of experts model, Mistral have really driven just tons of evolution. You see lots and lots of models getting closer and closer to the big private models.

Starting point is 00:10:45 And so I think the set of tools has never looked better. And yet, if you think about where I think it could be, I think there is so much research coming out. It's hard to even comprehend. I've read more research papers in the past year by a factor of 10 than the whole rest of my life because the research is absolutely breakneck. So I feel totally confident

Starting point is 00:11:05 in saying the best is yet to come on both those accounts. And so you start thinking about what this means for us day to day, right? What is it going to look like when an AI can independently go and introspect all your cloud infrastructure, turn it back into CloudFormation templates or Terraform's configs that you can just go and apply on a new provider or even swap between things like Terraform and CloudFormation. You know, what does it look like when you can say, I need an app that does this and you walk away and come back 12 hours later and it's, you know, looked up hundreds of web pages, read docs, you know, coded for hours and hours in a sandbox, it's done.

Starting point is 00:11:42 You don't have to touch anything because it can see the screen. It knows what acceptance criteria look like. We are not to that place yet, but I think that's low hanging fruit. Like this is not the far distant future. This is something that's coming very soon. So I think it gives you like a broad overview of how crazy I think things are going to get. Yeah. I want to zoom in on RAG for a second because that came up last week and it's come up a few times here. And frankly, I want to zoom in on RAG for a second, because that came up last week, and it's come up a few times here. And frankly, I'm really excited about that, because I think retrieval automated generation is how stuff should be done. I'm sorry. If you're just doing generation, where does the data come from again? I mean, don't you want people to go look stuff up?

Starting point is 00:12:23 I mean, that's what my teacher told me. How do I, you know, how do I know something? Look it up. You know, it seems as that just becomes how stuff is done instead of a specific trend. That's just sort of the right way to do it. Am I off base here, or do you think that that's just one more trend? No, I think that you're spot on. I mean, I will say this. I think that I'm going to go devil's advocate in a small area, but then I'm going to bring it back. The truth is we are very, very, very naive, like across the industry about how to kind of incrementally and easily fine tune a model, right? So the idea that you could take ongoing information and kind of slowly blend it into an existing model or a fork of an existing model, right? That could be specific to a group or a user or a use case, et cetera,

Starting point is 00:13:27 inside of a specific enterprise. We're nowhere near, you know, tapping out on that. In fact, people are really far. That's a further step down the journey than rag. And so we may see some rag use cases shift to that. And the reason why is because if you can integrate the knowledge or the base model, then if you can accurately generate and you don't have to go do lookups, you're basically fine tuning it once to get that knowledge integrated. And then you get all the inference is exactly the

Starting point is 00:13:56 same forever. Inference cost doesn't change once you've blended that knowledge in, but it's trickier than it sounds. Now, the reality is you still need RAG, because the truth is there's a limited amount of information that a model can hold. And that's especially true for smaller parameter models when you fine tune them. And so being able to kind of tap into any specific thing and just use it on the fly, it's always going to be a thing to some level, right? Whether it's about freshness or whether it's about the long tail of data, you know, having talked to people who have over a hundred petabytes of data, there's no way you can't meaningfully fine tune that into a model. It could never learn it. It's just not capable of holding an information today. But there's a really interesting angle because, you know, retrieval augmented generation, we know comes from taking a lot of documents and

Starting point is 00:14:43 information and we generally make it retrievable searchable right classic ways you create embeddings but those embeddings in a vector database one of the kind of naive look how early days it is is like this growing evidence that you know vector database is not all an embedding is not always the right way to do things like classic pm25 you know bag of words reverse index searches can be better. Sorry to get all crazy technical. But one of the things you realize though, too, is that RAG is a very close cousin of memory, right? And having an AI be able to do complex things requires you to prompt it over and over and over again. Like speaking of Devin, right? This engineer, well, it's off coding for 16 hours. How many times do you think it runs an LLM? Thousands, tens of thousands, a hundred thousand.

Starting point is 00:15:30 It's really kind of unbelievable how many times you're going to have to prompt it. In fact, even if you have a fairly simple query, you actually get much better results. If you can prompt the model over and over again, you'll say, give me your top three answers to this question. And then you'll go back to it again in a totally new conversation with no history and say, which of these three best answers the question. And then you'll change the order of all those things, right? There's a lot of techniques, but they require you to kind of go back to the well over and over again. All those kinds of things require you also to store those kinds of intermediate memories, right? If you say, go code something and you want an agent to come up with a

Starting point is 00:16:04 plan and then flesh out each step of the plan and then execute step-by-step, that requires a lot of data. And that's a form of RAG. You're kind of generating it with the model and then also retrieving it. Then of course, there's like the classic RAG, right? Where you're going and looking up customer records or FAQs or technical documentation, which is immensely valuable, honestly, because AIs are really good at sifting that out much better than us kind of searching bad search pages. So that's powerful all in its own right. And yet there's like so many different ways it's going to branch out. How all of these objectives across line of business and IT start hitting things like privacy policies, ethics policies, different things that usually aren't associated with the core of IT. around how to implement these new technologies while being mindful of some of the decisions and guardrails that they put in place and how their organization holistically runs? I'm going to say I love that question because, honestly, I don't feel like it gets asked enough. And one of the things that I saw was that two years ago, one year ago, even before we founded Kamiwaza,

Starting point is 00:17:29 I saw a lot of enterprises really struggling with the exact questions you're asking. It doesn't even need to be generative AI. If you want to have a data scientist and data analyst, there's data all over the place. It's in disparate catalogs or sometimes no catalog. You know, sometimes it has a schema, but sometimes it's a parquet table that's literally labeled field one, field two, field three of the columns. And some poor data scientist is literally trying to disentangle what it means without having any sort of like reference answer key. And I think those are big challenges. Access control is a huge problem. And it's a little bit intermediated if you're purely at the cloud, because a lot of people, you know, find some easy cloud success, just leveraging like IAM type, you know, functions to control access. Databricks does

Starting point is 00:18:10 really well there too, right? Snowflake does really well. But when you start getting into these cases where you have information that's all over the enterprise, and there's sort of different rules, different access controls, different security levels, sometimes it's in different countries, there's different regulatory regimes that apply to it, right? Those become very, very large challenges. And I think at Kamiawaza, that was something we were really keeping in mind while we were thinking about our distributed data engine. The fact that you as a developer, as a data scientist, you don't want to wrestle with those things. And even if you know the answers to some questions like, who should have access and can we move this data out of the country? Is there a tool set that really makes it easy to implement that? It's not great. And

Starting point is 00:18:48 certainly there's nothing specific to generative AI. And I think that really drove us to think about how we can make that easy. I'm really passionate about seeing enterprises be able to adopt this technology. I was using AWS in 2007, right before EBS volumes existed. And it's like, if I try to play that timeframe for the slow adoption of cloud on Degenerate AI, I just feel like humanity is missing out, right? And so we're really driving to make it easier because honestly, I want to see enterprises running trillions of inference calls so that we can just automate all the kind of grunt work of our day and get on to doing bigger and better and more powerful things over time. It's interesting that you bring up the question of sovereignty, though, when it comes to data. What about model sovereignty? I'm curious

Starting point is 00:19:36 about how that's going to hit us in the future. We've already seen a lot of instances where companies, you know, I'm not trying to throw stones at them, but some of these AI companies have been less than forthcoming about the training data that they used. end up having to use different models in different jurisdictions based on the laws of, for example, the EU versus the US versus Saudi Arabia or someplace like that. I mean, very different laws, very different legal scheme, and very different acceptance of training data. Yeah, that's such a fascinating question. And I don't even know, by the way, that I've never fact-checked this outside, but I did have a colleague that was doing a lot of work on AI for a particular use case way pre-generative AI. This is more classic, although it was still neural network driven. And he actually told me that to deploy their model into China, they actually had to train it in China. And they actually had a whole separate training process to also train the model in China.

Starting point is 00:20:42 What we see going on right now with the AI Act being passed to the EU, another fantastic example, right? And it's frankly, very early days. I think that there are going to be some very serious pitch battles over everything from, you know, model safety to regulatory regimes that need to be in place for all kinds of, you know, risks, real or perceived. I think the question of privacy, the questions about copyright, which is like really dramatic, gets a lot of press,

Starting point is 00:21:10 probably kind of hashes out to not a lot, I think, but it's still, there's a lot of risk factors. Even beyond those things, which are the big like prime mover, like nation state problems. It's like, one of the things that we bring up is like, let's say you deploy a model off of Hugging Face. Let's say you wake up tomorrow and somebody replaced a file, changed the config.

Starting point is 00:21:29 Maybe they thought it was a bug. Maybe they changed it for their own reasons. But then your whole stack collapses, right? And like that's one of the things that we tried to solve really early on was kind of providing some of that operational stability. But I think it's a great question and it's one of those things you don't necessarily spend a ton of time thinking about when you're in that, like, let's kick the tires POC phase. But I think that number one, people are really going to end up, you know, wanting to fine tune models of their own as they kind of grow in maturity, right? Because it's going to fit their company, their use case, whether it's tone, whether it's the guardrails, whether it's the use case, you could just get so much

Starting point is 00:22:05 better performance. Like fine tuning for a specific use case makes models almost superhumanly good at that, right? And so you can actually surpass something like a GPT-4 when you're training on that use case. We're going to see a lot of that. The skills are just not there for people to do that at scale yet. But then you get the win, right, of you're going to be able to do much cheaper inference at larger scale, have full control, you know, actually derive, you know, the changes that you want to see in the model behavior. And so I think guaranteed,

Starting point is 00:22:34 you're going to want to see some of that model destiny. But then to your point, as you have that model, there's going to be questions of, are we getting it commercially? Is it open source? What do we have to do? You know, you, even as a large enterprise, if you're multinational, you may going to be questions of, are we getting it commercially? Is it open source? What do we have to do? You know, you, even as a large enterprise, if you're multinational, you may have to have different copies of the model that are trained in different ways with different data because

Starting point is 00:22:52 you're not allowed to use data trained on X, Y, and Z when you put the data in the EU. Like a great example there, will you be able to train on data that you got based on the customer records in the EU and then push the model over to the United States? Even if you say, oh, it's not personally identifiable, you could see a regulation where they say, no, no, it's not safe because you might accidentally leak data that would be protected, which is very easy for me to picture them making that claim. So yeah, great question. Definitely a lot of challenges and a lot of unknown unknowns ahead, I think, in that arena. When you were talking, it made me think of, if we have multi-jurisdictional models, do we get into regional truths based on different AI models?

Starting point is 00:23:35 And that gets very dystopian very quickly. When you look at 2024 and you look at some of these topics that we're talking about, things are moving so fast. Do you think we're going to be still in the same place of enterprises looking for implementation and still, you know, seeking those tools, seeking the path forward? Or do you think that we're going to quickly move into this feels old hat? I think both because that whole bell curve concept, right, there will definitely be some laggards and we'll actually be having that like much as we said, cloud leader, cloud laggard, you know, so many years ago, that same kind of paradigm is going to apply. On the other hand, I would bet, you know, dollars to donuts that there's going to be a couple of things that just drop your job this year. Like, you know, the figure being a robot being powered by open AI.

Starting point is 00:24:23 I knew that was coming because I follow them both really closely. But I think that type of thing where you go, wait a second, like what is really possible here? And your mind just kind of explodes. We're going to see a few of those things. We're also going to see just a really huge rank and file improvement across the board, you know, in frameworks, in tooling. We're obviously trying really hard to contribute to that at Kamehwaza, but I think that there are just so many tools and so much research. And actually, so many things

Starting point is 00:24:50 that I've read about in research papers aren't really, quote unquote, in the wild yet. And so I think it's going to be a breakneck year. Do not expect it to slow down for sure. But by the same token, you're going to see the maturity extend and advance quite a bit. I absolutely agree with you, Matthew. I feel like the world is going to change dramatically in 2024. The thing that

Starting point is 00:25:13 strikes me is, to your point, that we spent 2023 sort of playing with a new toy of, you know, let's see what we can do. Let's see the cool stuff. I mean, I'm not seeing as many people posting on social media, look at this funny thing OpenAI wrote, right? Look at this funny thing ChatGPT wrote. Look at these funny pictures as sort of an end discussion. Now, I think what we're going to see is people say, okay, that shiny new toy, you know, we've played with that now. Now, let's see where it goes from here. And I do think that there's going to be some radical new use cases. I think it's going to be implemented everywhere. And as I said to Paul last week, and I think that what I'm hearing from you too, is that these models and frameworks are going to be deployed all over the place in

Starting point is 00:26:03 2024. And we're just going gonna see bang, bang, bang, new applications and new use cases for AI this year that are gonna just blow us away. And that's pretty cool. Pretty excited to see where that goes. So thank you so much for joining us on the discussion today. It's been a short discussion.

Starting point is 00:26:21 I mean, this is a short podcast. Where can people continue this conversation with you? There's lots a short discussion. I mean, this is a short podcast. Where can people continue this conversation with you? There's lots of great places. I myself do a podcast called AI Every Day, and people are welcome to check it out. I'm on LinkedIn. You can follow me on Twitter at Matt Wallace is the handle. So it's easy to find.

Starting point is 00:26:39 Of course, please check out our website, kamiwaza.ai, which is fantastic. And I'm also going to be presenting on the topic of enterprise AI at SW2Con, which is from the organizers of GlueCon. It's this May. In fact, you can use the code SPEAKER15 to get early word registration if you're catching this early enough, 15% off that. However you find me, I'm pretty easy to get a hold of, but reach out.

Starting point is 00:27:04 I mean, I definitely do love talking about this. I like to tell everybody, this is the thing that wakes me up at five in the morning and I can't sleep because it's just too exciting. So I feel really privileged to kind of find something that is so awesome and so motivating and exciting for me personally, right as it seems to be the tsunami that's just going to wash over the industry. Absolutely. Absolutely. And of course, we got to hear a presentation on Kamiwaza at our AI Field Day event, which Allison and I were both part of. So I would recommend people search your favorite search engine for Kamiwaza and AI Field Day, and you'll find those videos. Otherwise, you can go to techfielday.com to see them. Allison, what's new with you? I'm still publishing on the techarena.net, Steven, on AI, 5G, and from the cloud to the edge and everywhere in between. I'm going to be at GTC and at MemCon.

Starting point is 00:28:05 So you can check my page out for the latest in terms of industry innovations. Thanks a lot. Yeah, I do enjoy the podcasts as well that you record with notable people in the industry. And I urge people to check those out too. Thank you for listening to Utilizing AI, part of the Utilizing Tech podcast series. You can find this podcast in your favorite applications as well as on YouTube if you're interested in seeing the video.

Starting point is 00:28:33 If you enjoyed this discussion, please do leave us a rating or a review or a comment. This podcast is brought to you by Tech Field Day, home of IT experts from across the enterprise, now part of Futurum Group. For show notes and more episodes, head over to our dedicated website, which is utilizingtech.com, or find us on X Twitter and Mastodon at Utilizing Tech. Thanks for listening, and we will see you next week.

Utilizing Tech - Season 7: AI Data Infrastructure Presented by Solidigm - 06x10: Using AI in Modern Applications with Matthew Wallace

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.