Utilizing Tech - Season 7: AI Data Infrastructure Presented by Solidigm - 2x02: The Last Mile - Connecting AI to Enterprise Applications with Monte Zweben

Episode Date: January 12, 2021

There are many “last mile” items on the enterprise checklist, and companies are struggling to connect everything together. In this episode, Monte Zweben, CEO of Splice Machine, discusses feature stores with Andy Thurai and Stephen Foskett. Data engineers maintain data pipelines, data scientists maintain the data store, and machine learning engineers are trying to create models and package them so they will be useful. One idea is to store a model in a relational database, store records in a feature table, and enable the database to trigger a model based on this data. That’s what Splice Machines is implementing - in-database ML deployment. SQL is making a comeback in ML, with scale-out solutions providing a more familiar and usable environment than leading noSQL databases. Monte believes that SQL will be the dominant data paradigm for machine learning, modeling, experimentation, and deployment. After all, SQL is the dominant language of enterprise data scientists. Guests and Hosts Monte Zweben is CEO of Splice Machine. Find Monte on Twitter as @MZweben and Splice Machine as @SpliceMachine. Andy Thurai, technology influencer and thought leader. Find Andy’s content at theFieldCTO.com and on Twitter at @AndyThurai. Stephen Foskett, Publisher of Gestalt IT and Organizer of Tech Field Day. Find Stephen’s writing at GestaltIT.com and on Twitter at @SFoskett Date: 1/12/2021 Tags: @SFoskett, @AndyThurai, @MZweben, @SpliceMachine

Welcome to Utilizing AI, the podcast about enterprise applications for machine learning, deep learning, and other artificial intelligence topics. Each episode brings together experts in enterprise infrastructure to discuss applications of AI in today's data center. Today we're discussing the last mile. Basically, how do you operationalize AI in the enterprise? First, let's meet our guest, Monty. Hi, I'm Monty Zweden. I'm the CEO of Splice Machine,

and I'm really excited to be here today and talking about that last mile. And I am Andy Thorai, co-host of this podcast, founder and principal at thefieldcto.com, home of the Unbiased Advisory Services. Check us out at thefieldcto.com and on Twitter at Andy Thorei. And finally, I'm Stephen Foskett. I am the organizer of Tech Field Day and the publisher of Gestalt IT and the host of this podcast. So you can find me on Twitter at S Foskett. So Monty, this is really right to the heart of what we've been talking about on utilizing AI since the very beginning. And essentially, how do you bring enterprise, how do you make enterprise AI real? And one of the things that we've been talking about is feature stores, and you're literally writing a book on feature stores. So talk to us a little bit. What is it that the enterprise needs in order to make AI a real thing?

Well, that's a great question. And that last mile is fundamentally about model deployment and creating repeatable training of models, fast serving of models, and lineage and traceability for governance. And I believe that these are the enterprise requirements, but there's one really important piece that has to be woven through those requirements, and that is scale. Many of the approaches to operationalizing that last mile has created a loosely coupled architecture. And what I mean by that is that the data layers and the machine learning layers have been separated. And what our philosophy is, and I think many people's philosophy in this space

is starting to merge the machine learning capabilities with the data layer. So a feature store that can, in one component, be able to serve features to models in milliseconds, and at the same time, be keeping track of features that are changing over time so that you can train models and be able to keep that consistent and also make that reusable through the enterprise is becoming a critical component that's necessary. That's an interesting point, right?

You talk about the last mile of the AI or ML modeling. So that's not all that different than the last mile of enterprise obligations that enterprises used to have. If you were to look at it, things like governance, security, compliance, making it production grade, there are so many issues that get solved in that last mile. Basically, you're taking an innovation piece. How do you make it production-ready kind of thing, right? So AI is kind of moving into that space. But then again, I see a lot of companies, they're still struggling with the, maybe I can call it as the first mile of innovation with AI.

They're struggling to create the proper models. I mean, do you see a lot of companies already starting to move into the last mile and or what time or in what phase of their innovation should they be thinking about moving towards the last mile? Yeah, that's a really interesting question because it's really all about the maturity curve. Almost every company has a couple of data scientists now, major fortune, let's say 500 companies, global 2000 companies. They have a few data scientists, even if they're very early on their evolution on the journey to AI. Those data scientists now can capture data from data warehouses very easily. They can build models very easily on many people's platforms, my company's platforms, many platforms that are out there from other vendors and from the cloud vendors themselves.

Then when they go to put something into production, when they're taking a model out of the lab and really want to weave it into the enterprise workflow, then everything becomes very difficult. Because now you've got this blend of persona that need to cooperate with each other. You have data engineers that are maintaining all the data pipelines. You have the data scientists that are more mathematically oriented that are building models and making sure they statistically converge with predictive accuracy. operations and trying to package up those models so they're useful for the operational applications that they're going to impact. And right at that first deployment, they start to realize this is a lot harder than I thought. And then they get that into production and they get to their second model deployment and they're like, oh my God, this is never going to scale

across my enterprise if I keep doing it this way. And so're like, oh my God, this is never going to scale across my enterprise if I keep doing it this way. And so that's when it becomes apparent that model deployment and feature stores and the whole MLOps world becomes something that's not a nice to have, it's a must have to scale. And I think they feel that pain upon that first model deployment, Andy. That's actually a really key point, I think, because I think a lot of AI applications come in sort of as one person's idea over here in the corner. It's kind of a skunkworks project, or it's something that that guy knows about, or the data scientist, she put that together. I'm not sure. It's all about basically

operationalizing these things and making them not just function, but making them scale, making them part of the overall enterprise application infrastructure. And I think that that's really the, that is the last mile, am I right? Yeah, I think so. And this is best maybe explored with a real example of how you can architect to facilitate that last mile and bridge some of these gaps. As I mentioned earlier, people have separated the

data layer from the machine learning layer often, but what happens when you merge them? And as an example, in terms of model deployment, a traditional approach to model deployment would typically be an endpoint deployment. A data scientist says, I've got this model. I think it works. It has a pretty good F1 score area under the curve metric. And I'd like to put this into operation. And they have to go get someone else usually to wrap that model with a RESTful API probably, and maybe package it as a microservice and potentially containerize it, push it out as an endpoint. And those are traditional software engineering practices that are necessary to do that.

But an alternative approach is to literally put the machine learning model into the database. What does that mean? Well, imagine if you had a low code way of saying deploy this model as a data scientist, maybe a Python function or even a click of a button on a UI. And imagine the machine learning framework able to take that model, store it in a relational database, a traditional relational database, meaning it's in SQL, but a scalable relational database. And there are many on the market. And imagine if you could deploy a model now by simply doing the following, insert records into a table, let's call it a feature table where each of the columns of

that table reflect features of the model. Perhaps there's some metadata in that table too as to who's asking for a prediction, maybe a timestamp for the time. And imagine if the database was able to recognize, wait a minute, there are new records in this feature table. Let's trigger a model and let's figure out what the predictions are, the inference, put them into the records with the features and automatically memorialize that just as a relational table. And now users of the model would just be accessing SQL to get those predictions. There isn't a container. There isn't orchestration.

There isn't a RESTful API. There's no creation of that endpoint. It's simply low code, one line of code, deploy, and now you can just query a database with those predictions. This is the future, I think, of the last mile. This is, we call this in-database model deployment. This is, but this is something that I think is going to become standard in the marketplace as you look at the maturation of machine learning and data layers. But what this does is it facilitates that last mile where you don't need as many special skill sets or special processes, and you can rapidly deploy. And you have governance, too, because databases are extremely well controlled.

There's access controls and backup and resiliency capabilities. So I think this facilitates the maturation of the deployment of machine learning models into that real-time fabric of the enterprise, but with governance, with controls, but without having to have all these moving pieces in a technology that people know already, which is SQL. So that's an example of, I think, how this space will mature. There are like 20 points you made in there. I want to double click and drill you on, but let's start with the first one. So you were talking about using a SQL for the digital enterprise. That's almost like, you know, it's like you touched the kosher word, kosher thing that all the digital enterprises are moving away from SQL because to no SQL and other, you know, time series and whatnot, because the sequels become this, you know, only monoliths can handle it in the back of the enterprise.

It doesn't scale all of those issues. Right. scale, all of those issues, right? So now you're saying not only that you could have the digital enterprises use, you know, equal and off that, you know, the SQL database itself, but also you're suggesting the AI ML models can use that even as a future store or one-click deployment, which sounds too good to be true. It sounds too good to be true. Okay, tell me about it. But the fact is, if you look at the trends, there definitely was. There was a move away from SQL, especially as, you know, Google published the MapReduce paper and the big data architectures

from the Hadoop world emerged, there was a tremendous push into a different data model. But if you look at the trends, SQL is back and SQL is big right now because the other languages just simply were not representative enough or comprehensive in their behaviors to really support what's necessary for what we're trying to achieve with AI. And if you look at the trend, in the beginning, we saw MapReduce as being a way to scale. And it was a tremendous paradigm breakthrough. It allowed mere Java programmers to be able to handle thousands of computations in parallel without having a PhD in distributed systems. That was really important. But then quickly thereafter, we started to see things like Spark come into the picture, Apache Spark,

where you could not have to work with these esoteric mapping and reducing functions, but just work with sets of data and apply transformations to that data. Much more, I would say, programmer friendly at that point. But then what happened? Every piece of the big data architecture needed SQL. Everyone, including Google, built SQL stores. What am I talking about? On Hadoop, it was Hive first. And then there was Impala. And of course, there's Spark SQL. In the Google world, there's BigQuery. And then there was Spanner. There were even systems before that. And if you look at the leading capabilities now, and even look at the tremendous successful

IPO of Snowflake on the data warehousing side, this is all SQL that's returning. So what's happened? There was a major turn away from SQL, but now people are coming back and it's back to stay. What happened was exactly what you said. The flaws of the first generation of SQL databases have been solved. And those flaws had to do with architecture, not SQL, but the architecture of the database. One piece of it was the fact that the database was centralized. It worked on one server pretty much or a loosely coupled set of servers. It wasn't a scale out architecture. It typically was a scale up architecture.

That's been blown out of the water now by a number of the relational database and data warehouse vendors out there, including Snowflake, including Spice Machine, including Databricks, including Google, and many others. But that just now gives, that just gives you infinite scale. And it auto shards the data so that you don't have to worry about what server you put your data on. It's spreading the data out, distributing it out in an efficient way. So that took away, I would say, at least 75% of the criticisms of the old relational database

model. The other piece of it had to do with flexibility of representation, changing columns, changing schema, and so forth, being able to provide even unstructured data in storage. All of the modern databases have made it very easy for you to change schema and to be able to be elastic where now you can turn things on and off. Imagine having a petabyte of data inside of a data warehouse and you just turn it off when you're not using it. Now we all can do that. So yes, I would argue SQL is back.

And the reason why it's back is because it is the most comprehensive data language. It does support transactional and analytical workloads in very big ways. It's been around for 40 years, so it's 30, 40 years, and so it's really been made robust, but now it truly scales. So the answer to your first question is that I think you're going to start to see much less of the schema on read or the no SQL world being dominant. I think SQL is here to stay. And yes, my second point, and we can talk about that, I'll let you guide our conversation, is now not only is SQL really powerful as a data platform in general to power apps or to do insights for analytics, it is going to be the foundation of the machine learning world.

It is going to be so tightly knit with the modeling environment, the experimentation environment, and the MLOps environment. And I think you'll start to see this from many different capable vendors out there merging these worlds. Let me ask you a follow-up question on that before we move on. I get that. I mean, look, SQL, there's a reason why Oracles and DVDs of the world have been our own and even charging an arm and a leg, right? A ton of money for the enterprises.

And enterprises completely rely on that. I'm not questioning that. But the AI and ML models seem to be a force fit into SQL, but we can have another podcast talk about in detail. But are you seeing really the major enterprises starting to use SQL for all of their AI ML modeling? Particularly, I'm asking this question because when you do an edge distribution of the models, it becomes really complicated. And how do you store and distribute that when you use the SQL database? What percent of the enterprises do you see is moving? Okay. So that's a great question. So on deployment

and in terms of the proper architecture for the deployment of a model, and I made the argument that in database deployment of a model is very powerful. You brought up the edge and the edge is an alternative case, especially if you're talking about an edge that's disconnected from the internet, at least intermittently connected. Perhaps you're on an oil rig and you have some disconnected capabilities. Even there, you can put racks of computers. So that's not really an edge. But a driverless vehicle might be a good example of the edge where you don't have the time to get to the cloud or you just for safety reasons can't rely on the cloud. I was talking to another customer about cruise ships as an example where there's spotty connection.

In cases where you're not connected, you do need to have a loosely coupled scenario where you may have an endpoint deployment and push that out to the edge. And then as the model is making its inferences, that gets queued up for essentially transfer to a cloud repository that can be the feature store and keeping the historical set of features that have been used so that you can continuously retrain offline or in a centralized location. So I do think there's a role for edge deployment. But on the other hand, and this is an observation, in financial services, in healthcare, even in process plans, petrochemical, oil and gas, energy, things like that, the large majority of deployments have access to the data store all the time. And so

while there are edge exceptions, being able to leverage your data store inextricably linked to your inferencing is something that I think is way the majority of the case. And edge point deployments, edge deployments where you have intermittent connectivity is a very specialized case, and that you have to accommodate that case. And I think it's important also to remember that AI is still a nascent product category in the enterprise, and the vast majority of enterprises and data scientists and data analytics folks, well, SQL is their native language. So if you are a data scientist at a major corporation, you worked there a long time, you're starting to think about bringing in AI or ML into the database, you're trying to think about how you can leverage that to better use your data. SQL is just what you speak. And so by having a SQL-based ML solution, I think that's a lot more attractive

in the enterprise than it might've been maybe in the academic space or in the hyperscaler space. Well, that was certainly what was on my mind when we started our company. And we can even talk about this at an even more extreme level than just the skill sets. In one of our customers' circumstances, a very interesting situation where they're trying to modernize an old application. It's an insurance company. They've got a robust insurance application that handles many insurance functions. In fact, all insurance functions, including underwriting and claims and disbursements and so forth. And this was an older application. It was originally written in a legacy language on a mainframe, and they've converted it to a modern

language. And they're moving it to a cloud. They're on a cloud migration. And the question was, can we incorporate AI into that old app? And one of the things that we tried to do, even more than just support SQL, was to try to support legacy SQL. So that companies can not just be relegated to writing new apps in SQL for AI. But imagine if you could take those legacy applications and literally run them on a different SQL database than they were originally designed for. They're probably designed for IBM DB2 or perhaps designed for Oracle. Imagine if you could easily migrate them over to a scale-out SQL system

and still run that old app with the same functionality without the rewrites that have been required over the years. And now on that scale-out platform, start injecting models into it where you can just simply add a few more pieces of logic in SQL to that application to leverage that AI, but without having to re-architect or rewrite that application. So not only is SQL prevalent in the skill set, but SQL is prevalent in billions, if not trillions of lines of code out there that run organizations everywhere. I believe that those applications are not going to remain static. and bringing them to a more modern architecture, you'll start to see enterprise applications moving off of older frameworks, maybe even mainframes, into the cloud and actually being replatformed onto scale-out SQL, and then with AI models being injected into them. In fact, we're banking on that as a part of our business model.

And I think that you're seeing in every SI organization, significant practices on application modernization, cloud migration and application modernization. That's not just about bringing something to the cloud. That's about bringing an application to a modern architecture and then extending it and enriching it with AI. And I think it's very exciting times because we don't have to wait for people to build new applications to make those applications smarter and intelligent with machine learning models. You literally can inject machine learning into old applications much easier than ever before. I saw a couple of use cases you guys do using machine learning, particularly the one that you mentioned about doing a live leak detection. Tell me a little bit more about that. That's

very interesting because obviously places like water and chemical plants, especially when humans cannot go, this could be a huge use case. How is that applicable in the industrial environment? I'm really excited about industrial problems that we've most recently been applying some of our technology to. I've been exposed now to problems in petrochemical plants like ammonia plants, desalination plants, and other industrial frameworks where there is an order of magnitude more data than let's say in a bank or an insurance company or healthcare company. You've got these distributed control systems, DCS systems that are generating readings from sensors at significant velocity at great volume. Maybe in the larger plants at a million tags where a tag is a value of a sensor reading at a particular timestamp, maybe a million tags

a second. Okay, that's big. And typically, this data flows out of DCS systems into what are called historians. And these historians have been kind of the old school data store that is analogous to the SQL data stores in the business systems. Historians are the time series databases of the industrial systems. And we really haven't made a lot of progress of being able to inject AI into these industrial scenarios like leak detection or maybe mechanical failure, or even just doing some predictions about load distribution and utilities, because the data has just been too large. And these historians haven't been able to scale in a way that provides the ability to do both the storing of time-stamped time series information,

as well as the analysis on it. So the next generation of historians, I think will look more like the big data architecture, where you're able to ingest very large amounts of data, SQL databases like ours, or even key value stores that have been used in the big data architectures, things like HBase and other time series databases, Influx and Prometheus as examples. Some of these systems now can handle tremendous scale of time series data, but they'll be very tightly coupled in our case on the same platform, in other cases, through communication with analytical frameworks that can truly scale. So now you finally have the ability to be able to ingest

data from pressure, volume, and temperature sensors at grand scale, be able to, you know, do, you know, trending on that data in split seconds, be able to run that through models in situ and literally provide an operator with a dashboard that not only has real time sort of panels of what the sensors are reading, but right next to what the sensors are reading are predictions of failure states, predictions of

leaks, predictions of remaining useful life, predictions of mechanical failure, so that the operator is now not just looking at dials and at real data and trends and saying, oh, I think something's out of the ordinary. That kind of anomaly detection is literally now proposed right in front of the operator on screens because of that real-time stream of AI that's going on. This is a new world. And this is happening quickly. There are a couple of companies that just went public, or one is going public, that's C3 AI. That's led by Tom Siebel, who was on a couple of my boards in the past. He's a great mentor of mine.

Palantir, very well-known organization. Those are kind of black box, big framework approaches to this problem. We take more of an open source SQL data platform approach that's just much more modular and open source oriented. But what we're all doing is putting this big data architecture into the industrial frameworks that exist out there and providing that real time AI. think it's a very exciting time. I remember when we first did this back at NASA. I didn't mention some of my history. I helped co-run NASA's first AI lab. And I remember when mission control went from green screens to intelligent expert systems. I remember when my project took scheduling systems for the refurbishment of the space shuttle from little pieces of magnets on big boards and rooms to try to get people to prepare and turn it into real-time planning and

scheduling of refurbishment of the space shuttle. This was a major change that took place in the space agency. The Defense Department has seen that. Now industry is experiencing that with real-time AI. Applying AI in the industrial world or even real world, that's actually a good kind of trying to solve some problems, but also it goes to an extent beyond at times. For example, I was reading this, one of the, either a Scandinavian country or a Baltic country, the traffic was so horrible in the mornings around from seven to nine. So they wanted to stop that. So they put an AI algorithm and then based on that, they were charging tolls, which could be 10 times as much as a normal toll they were charged to go through the city center,

right? And then the reverse of the problem had to happen because people, when you force them, don't go out there, they won't listen to you, but you charge them 10 times more. And they were trying to recruit people now because nobody was driving through the town center through seven to nine in the morning. So yeah, I mean, predicting the patterns and usage and making recommendations in the real life scenarios, whether it's industrial

traffic or whatnot, that's where AI need to come up to speed fairly quickly to solve the problems that we can't solve otherwise. Yeah, I have to agree. And is nothing more satisfying than seeing some of this technology hit real-world problems, helping people with very tangible issues. One of our customers is using a real-time AI model to predict the trajectory of neurological disease and using population data across networks of neurological clinics to try to predict what's happening to a patient who has multiple sclerosis to get not just what one doctor thinks, but what the whole network of doctors

have performed and the outcomes that have historically been observed and using that to predict the trajectory for a particular patient. I think we're going to start to see the applications here injected into real-time processes across every element that we are involved in, where there are human experts doing things. And we will see, in many cases, very positive outcomes for people. And of course, we have to be careful, make sure that things are ethically done and governed properly. It's all other conversation. But the new capabilities of merging data and machine learning together to deliver on predictive real-time processes is really emerging in a very real way in industry today.

So, Monty, I have the joy of doing something a little fun here at the end of the podcast. So for season two, basically, I've got a set of questions that I'm asking the guests. And I've got a number of different questions. I'm just going to pick a couple of them and quickly ask you. So this is the lightning round. And don't be surprised if you don't know the answer. Just make something up. Okay. So here we go. Here we go. Ready? Question number one. How long will it take for there to be a conversational AI that can pass the Turing test and fool the average person? I think that's already happened on the average person with a conversational AI.

I think that happened a while ago in some ways, but GPT-3, the newest model that has been talked about lately that's an extremely deep transcoder model can do that today. All right. Another question. In your opinion, does it drive you crazy? Are you nodding along in agreement when people use machine learning, deep learning, and artificial

intelligence as synonyms? This is, it drives me a little bit crazy because AI is a lot larger than just machine learning. And machine learning is a lot larger than just machine learning. And machine learning is a lot larger than just deep learning. So I think it's important to make that distinction. There's a whole host of AI techniques, planning, scheduling, reasoning, logic, non machine learning oriented approaches to natural language like computational linguistics approaches.

There's a tremendous amount more to AI than just machine learning. And machine learning happens to be the most successful right now. So I think it's getting the limelight. And clearly deep learning has been very effective on unstructured data. So yeah, I think we need to try to be a little bit more disciplined in our terminology. Where can people connect with you and follow your thoughts on artificial intelligence and other topics? Well, that's great. So my company is at www.spicemachine.com. I happen to have a podcast as well called ML Minutes, where practitioners of machine learning have to answer our questions in one minute or less, I clearly wouldn't have been a good guest on that particular podcast. And also,

I'm writing a new book called Feature Stores for Machine Learning to be published on Manning Publishers. And I'm very excited about that coming out soon. And if people have thoughts on feature stores, I'd love to hear them. All right. Well, thank you so much, I really appreciate it. If you enjoyed this discussion, please do remember to subscribe, rate, and review the show on iTunes, since that really does help our visibility. And please do share this show with your friends. This podcast is brought to you by gestaltit.com, your home for IT coverage from across the enterprise. For show notes and more episodes, go to utilizing-ai.com

or you can find us on Twitter at utilizing underscore AI. Thanks and we'll see you next week. Thank you.