The Changelog: Software Development, Open Source - Putting AI in a box at MachineBox [rebroadcast] (Interview)

Episode Date: July 11, 2018

In this special episode of The Changelog we’re sharing a full-length episode of our newly launched podcast called Practical AI — covering AI, Machine Learning, and Data Science. In this episode Mat Ryer and David Hernandez joined Daniel and Chris to talk about MachineBox, building a company around AI, and democratizing AI.

Bandwidth for Changelog is provided by Fastly. Learn more at Fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at Rollbar.com and we're hosted on Linode servers. Head to Linode.com slash Changelog. This episode is brought to you by Airbrake. Airbrake is full stack real-time error monitoring. Get real-time error alerts plus all the information you need to fix errors fast. And in this segment, I'm talking with Joe Godfrey, the CEO of Airbrake, about taking the guesswork out of rollbacks. So imagine a developer has found a serious problem and they're trying to figure out what to do, whether to roll back yes or no. And if they roll back, what do they roll back? You know, because they may have released 10, 20, or even 50 new commits that day, and

they're just not 100% sure which commit caused the problem. That's exactly where a tool like Airbrake comes into play. So instead of guessing and trying to figure out, well, this one seems like it's the code that's most likely to be related to what's happening. And first customer reported about 2 o'clock, so maybe it went out a little bit before that. Airbrake has this great tool called the Deploy the deployment dashboard that literally will show you every single revision, who did the deployment, and it will show you which errors were tied to each deployment. So you can pretty quickly figure out, aha, this error started happening only after this code was deployed. And by the way, it will also tell you which errors were fixed. So that if you go and go ahead and roll back a revision or try to put a fix or a bug, you can then look and say, did it actually get fixed or is it still happening and do I need to take different action?

I just know from my experience, we've spent a lot of time guessing about what to do and a lot of time arguing about whether those guesses were any good or not. And I would have loved to have just had some real anecdotal evidence that proved, no, this bug was caused by this release. We're either going to fix it real quick based on all the information that rate gives or we're going to roll it back. And we're not going to guess and roll back a whole day's worth of work just to make sure we catch this bug.

All right, check out Airbrake at airbrake.io slash changelog. Our listeners get Airbrake for free for 30 days, plus you get 50% off your first three months. Try it free today. Once again, airbrake.io slash changelog. Hey, everyone. I'm Tim Smith, senior producer here at Changelog. In case you didn't know, we recently launched a new show called Practical AI.

It's a show that talks about making artificial intelligence practical, productive, and accessible to everyone. Today, I'm excited to bring you an episode of Pract about building a company around AI and democratizing AI. Enjoy. Welcome to Matt and David from MachineBox. It's great to have you here on Practical AI. I know that Chris and I, when we started thinking about guests for Practical AI, and I was thinking about our slogan or our mantra at Practical AI, which is making artificial intelligence practical, productive, and accessible to everyone. I know the first people that came to my mind were Matt and David from MachineBox. So welcome, guys. Thank you very much. It's great to be here. Yeah, thank you. Yeah. Let's start out maybe with just a kind of brief personal intro from both of you guys. So David, why don't you start by giving us a little personal intro? Yeah, I'm David Hernandez. So my background is in computer science and engineering so I was studying back more than 10 years ago

in the uni machine learning and artificial intelligence mostly I never used it till recently the last three years that the machine learning was booming and I started to to get through implement refresh my my knowledge and I started to implement things. And probably that's how we started Machine Box in some way. And professionally, I've been developing since I finished my degree. It's almost also 10 years doing distributed systems, website. My highlights probably, I work atbc at 2012 for the for the olympics so

we were delivering the real-time system to all the stats and all the video player uh data for for the olympics basically uh was real nice project so so yeah that's it sounds great and matt why don't you give us a little bit of an idea of uh of where you're coming from yeah sure so hi my name is matt ryer i've been doing computer science all my career in various forms i spend a lot of time in the go community at the moment i kind of fell in love with Go as a language before it was released as version one. There's a little experimental tag in Google App Engine, and I wanted to build something on App Engine. And anything that's got a little experimental tag is going to grab my attention, it always has. So I jumped into the language kind of quite early. And I've just been kind of using Go since then, really, wherever I can.

And it turns out you can use it everywhere. I speak at conferences about Go mainly. And also I have a book, Go Programming Blueprints, which is nice because you build real projects in that book. So it's not a book where you learn the language or you just learn theory. You actually build real things so it's very practical and very pragmatic and that's why i quite like the the the way you guys are approaching this podcast because you know complicated things can be extremely powerful

but they're very difficult for people to kind of marshal and and get to get you know get into a shape that they can put into production and And that's really our philosophy at MachineBox is to give people a head start on that and get them into production much quicker. And hey, Matt, while we're at it, what's the name of your book before we go on? Oh, it's called, thank you. It's called Go Programming Blueprints,

colon second edition. Okay, thank you very much. Yeah, you don't have to say it in that accent, but it helps. It sounds much better. I think it does help. So how did you guys originally meet and how did you start thinking about forming a company together that's focusing on AI? Well, that's interesting story. So, um, back few years ago, uh, I was one of the organizers of GoLine UK, the first one. Now GopherCon UK or just GopherCon. I don't know. GopherCon UK.

GopherCon UK. So Matt was one of the speakers. So I met actually Matt in that conference. He was in another company before and I was looking kind of for a job. I was a contractor at the time so I joined the same company that Matt was and we met there basically. We worked there for a few years. Yeah, David has a really kind of unique ability to think very clearly about big problems that are um otherwise very complicated and that's that's a key skill for any team to have if you can bring somebody in that can look at these kind of big broad problems like massive kind of scale planet scale sort of

problems like you said like david mentioned earlier he he was part of the team that that delivered the software that ran the olympics and you can't there's no dry runs of that you can't say to everyone guys can we just like a week before can we just have another olympics just so we can test out all the uh all the software um they'd probably say no. So having somebody like that on a team is invaluable. And it was very natural when it came to looking at machine learning and David's expertise in it. It was kind of his eureka moment where he said, you know, we could actually kind of containerize this and deliver it in a way that makes it very trivial for everybody to use machine learning capabilities rather than having to learn TensorFlow and work in these kind of abstract mathematical models. We could tell some stories

differently. We could give people an API that just makes it very clear and sort of changes the way you think about the problem a little bit. It focuses really on the problem that you're trying to solve rather than technical sort of low-level machine learning components that you might use to solve it. Yeah, that's great. And again, I think, you know, I've been a MachineBox user for quite a while. I think, you know, very soon after you guys launched, I was super excited about it just because of the practicality of the project. So in terms of like what MachineBox is actually, you know, as a developer, like if I was wanting to use MachineBox to do something, what might that something be? And what would be the interface to doing that? Yeah, sure. So MachineBox is basically we deliver machine learning models

in Docker containers. So what you basically need is Docker installed in your computer that is available in any major major platform windows mac and linux you just docker pull one of our images you have a nice api in in our images so you only need to know about http um http apis to to get started and do, for example, face recognition. That is one of our most famous boxes. So you can add face recognition to your stack in just minutes. So that's basic tools. Docker, a little

knowledge to do HTTP APIs as a programmer that probably every programmer should learn that skill nowadays. And that's basically, you don't need any other knowledge. So just to clarify, I mean, really, like if I was a data scientist or a developer or whatever I am,

there's a lot of APIs out there, both, you know, from the cloud platforms, like with, you know, machine learning, but also other things like if I want to send an email or something programmatically, there's, there's like a REST API for, for that, which uses HTTP and, and, and JSON. And so you're saying kind of one of your goals is to really make the interface to doing something complicated maybe like uh facial recognition or something as easy as it is to you know send an email via via one of those apis is that is that kind of a good yeah that's that's exactly right yeah so essentially the machine learning learning that's going on inside the boxes is very complicated.

And sometimes we mix different kinds of technologies in different ways, where and if we try to explain how to do that, it would be very complex. And I think the deployment would be difficult. And even just managing the dependencies would be a bit of a nightmare. So we take on all that pain and provide APIs that tell different stories. So for example, you mentioned facial recognition. Facebox is a Docker container. You download it, you run it, you then have HTTP access and the operations you can do are things like, here's an image, tell me all the faces in that image and give me the coordinates of the faces. Not only that, if you recognize these people

and who the face belongs to, tell me who that person is as well. And then there's another API call to teach. And we support one-shot teaching, which is also pretty kind of rare still which is but it just means that with one image so daniel i could take an image of your face and teach face box with one example image and then if we took a big photo big photograph at a conference and you were in it face box would be able to find you and identify you.

So you get that facial recognition capability, and it's only a couple of API endpoints you have to learn. It's basically teach this face, and here's an image. Who do you see in there? And then, yeah, it's all JSON because we wanted to just feel really familiar and just fit into what people already had. And HTTP and JSON APIs still dominate. They're the simplest to use. You can use them like they're nice because you can just use them in the browser.

And when you run one of our boxes, we actually host inside the box, a little private website, which you access through localhost 8080. And that website contains all the API documentation, but also lets you interact with the box without even writing any code. Because it's very important on our mission to make, first of all, communicate what's possible in a very simple way, and then make that easy to play with and get to use so that people can see the power of it. And then once they've sold on that, then it's just a question of making that, making the integration easy and operations.

And so we're really focusing on that whole flow end to end. In particular, we care about people without any kind of machine learning experience being able to use these powerful technologies. So it sounds like MachineBox hasBox, you've taken the machine learning part and abstracted that and put it in a little black box for your end users. Who specifically are you targeting as your customer for this? Well, we have already paying customers. And so I say already, because although Daniel started playing with MachineBox way before we really

launched anything and one of the nice things about the fact the way we approach our developer community is we give them the technology for free early and let them just play with it and that process what happens is first of all any bugs are immediately found and squashed luckily it doesn't happen very often we do a lot of uh testing and test driven development and other techniques which help us when it comes to kind of code quality but beyond that we get to validate the way we've told a story and also you know if the apis really make sense for the particular way in which their system expects to use a technology like this. So we've had, we see customers of all kinds. We really only, it's a developer tool. So this is for developers to integrate into their platform. So by and large, all of our audience are

developers, but the people that really kind of have so far found it to be useful are people who they understand machine learning in broad terms, some of them. But they know that it's a lot of effort to go to, to build your own things yourself. And then, you know, if you care about the data not leaving your own network, whether that's on-prem or your own cloud, because we're just Docker containers, you can spin them up anywhere and scale them anywhere. It's, you know, you keep control of all that data. So it's people who, they understand, they have already a need, which is great. They've got a problem that they want to use machine learning to solve. And then they use our APIs to solve that problem. So they're basically developers of all levels. Usually, I mean, some of them are just JavaScript developers,

some of them are Ruby. We do have a Go SDK. So we have a lot of gophers, we have a lot of go people that are using it. So it's really that, that's who we target is basically anyone's a potential target but specifically we've seen traction in in developers who don't want to have to do all the heavy lifting of machine learning you just want to get something and get going yeah my favorite uh users uh are it's it's kind of a personal opinion and it it doesn't necessarily mean that it's right. So my favorite users are DevOps, people doing DevOps, basically. Because they basically love it, because they usually don't have time or willing to learn any kind of data science.

They want to solve specific problems. And they find Machine Box and ouri is really good and really productive for that so we we we get a lot of love from from from devops the best comments that we we hear is if from people doing devops like oh i have this problem i want to solve it quickly i want to deploy it quickly and it it is just the the perfect tool for tool for that kind of people. And yeah, pretty much. Yeah, that's great. I know personally, I can attest to, you know, just the quality of the models.

I know I actually kind of got into a little bit of trouble at a conference because I was showing Facebox and kind of one shot updating of the of the model. And and people didn't believe me that it actually worked, worked that well. So that made for. Yeah, that's happened to us as well. I think in a demo, we've had it where people just think we've spoofed it um yeah i know it's surprising because um you know you we're told again and again for machine learning to be any good you need massive amounts of training data so that's why um it's and and really the solution i mean it's kind of a bit secret of what we do but it's um which is just a clever uh use of technology inside the box, which allows us to provide that. But the thing is, we don't want people to have to worry about how it works.

We just want them to know that it works and integrate it and get to MVP really quickly. That's really another one of our goals. You know, a few weeks ago, I was in San Jose at NVIDIA's annual GPU technology conference. And through my employer, I was in a small group meeting with the NVIDIA CEO, Jensen Wong. And he noted something that I see you guys kind of going toward, and that we're really at a junction where software developers are becoming the targets of machine learning rather than just data scientists. And it will continue to be both. But he noted that that was a big strategic initiative on them was to target the software development community, which is somewhat new to these technologies.

And it seems that you guys have really centered your strategy around that approach. Yes. I mean, that's right i mean really what happened in if i'm being completely honest is we just built something that we needed to use we wanted to use some of these technologies and it's it's hard and we had constraints and you know some of the some of them at scale some of the prices of the the machine learning apis at scale really um it's it's really becomes prohibitive i mean it's it's just it's still quite expensive and still it's it's quite valuable i guess so that's why but we weren't we weren't really kind of too strategic about it in the beginning we just

thought let's just build let's build it how we think it should be built and how we would want to use it and from there we've then started to see uh traction and all and you know some great feedback from our on our developer experience This episode is brought to you by Linode, our cloud server of choice. It's so easy to get started. Head to linode.com slash changelog. Pick a plan, pick a distro, and pick a location, and in minutes deploy your Linode cloud server. They have draw-worthy hardware native ssd cloud storage 40 gigabit network intel e5 processors simple easy control panel 99.9 uptime guaranteed we are

never down 24 7 customer support 10 data centers three regions anywhere in the world they got you covered head to leno.com slash changelog to get $20 in hosting credit. That's four months free. Once again, leno.com slash changelog. And by GoCD. GoCD is an open source continuous delivery server built by ThoughtWorks. Check them out at gocd.org or on GitHub at github.com slash gocd. GoCD provides continuous delivery out of the box with its built-in pipelines, advanced traceability, and value stream visualization.

With GoCD, you can easily model, orchestrate, and visualize complex workflows from end to end with no problem. They support Kubernetes and modern infrastructure with Elastic on-demand agents and cloud deployments. To learn more about GoCD, visit gocd.org slash changelog. It's free to use and they have professional support and enterprise add-ons available from thoughtworks once again go cd.org changelog I kind of want to follow up a little bit on those, you know, that idea kind of that we mentioned around around the conference talks is, you know, you kind of use this machine box to do something. And it's doing something complicated under the hood and it's giving you great, you know, great results. But to some degree, you know, even though you might know generally what's happening

in the box, it still is a black box. And there's kind of a lot of back and forth in industry right now, at least in the circles that I kind of frequent around, you know, is treating machine learning and AI models as kind of a black box, a good thing or a bad thing. And, you know, AI, you know, like I can I can download pre-trained models and that sort of thing that I don't really understand right from the TensorFlow repo and other things. And often really they're, you know, I don't get the kind of results that are, that are, you know, either published results or the kind of quality that's promised from these pre-trained models. Now, the models that you're putting out are definitely, I get really good

quality, but I still don't really know all of what's going on on the inside. So in this case, like we're treating machine learning and AI models kind of like a black on on the inside. So in this case, like we're treating machine learning and AI models kind of like a black box. Why do you think in at least in certain cases, you know, treating models like this, like a black box can be can be a really good thing? Or maybe what what are what are some downsides or cases in which maybe you wouldn't want to treat them like that? Yeah, sure. yeah sure so yeah all the machine box models are kind of a black box so in that case we we don't have any explainability for any of the models but also most of the models are based in in neural networks so nobody has that

answer yet in the research there have some been research about it, but nobody knows what is happening inside. So you just mean in terms of the complexity of the models? Yeah, but also for use cases. or accept a credit or an insurance, it's quite important to understand what a model is predicting. I'm saying, oh, if my income is less than this quantity, the model is going to say, oh, you're not going to get the insurance or you're not going to get the credit. But for example, facial recognition, you care less about why the model is predicting

that this is matching a face rather than not matching this other identity so you you are more worried about the value that you can extract for that matching rather than the the value that you can get explaining why the model is doing so. So it's quite a balance and it really depends on the use cases. Mostly our use cases doesn't really matter the explainability. In most of the boxes we have, for example, classification box that allows you to build any kind of classifier, given text or images. So it may matter most for that kind of classifier given text or images so it may matter most for for that kind kind of

uh model but in general sense we we more focus on getting value for the models rather to explain what the models do um yeah that's that's a great point and i mean, to your guys point, I think, you know, if if you're not able to put your model into production and get any value out of it via a useful interface, then, you know, really what we're talking about is just, you know, AI research that isn't really applicable in a business setting. So you have to be able to get things into production. I think that's where this sort of black box treatment, in my opinion, is a really good thing in terms of providing a unified interface for developers and DevOps people and infrastructure people to interact with a model. Yeah, but anyway, it should be, I believe that the research is going to come through

and someday we can explain how a neural network does the reasoning and why a prediction is that prediction. So we probably try to keep up with the research and if that comes through, it's a possibility to add it to the boxes. Yeah, but those sorts of um those sorts of things and and a lot of the arguments against black boxing are for it's really i think people who are deep in machine learning they know about it um they want to um that you know they want to

invest time and resources into kind of building expertise and things like that. Lots of people aren't in a position where they can do that. So we, you know, we, we give them a capability. It's a solution. It's, it's, it's, they are models inside. Sometimes there are multiple ones inside each box, but there's also other things going on in there. So really is a solution that um you know we the only reason really that machine box isn't just completely an open source project is that it's just so complicated that it wouldn't be i don't think you know it's not like it's just kind of a trivial little little package that would be sensible to open source and everyone can get use out of to use to be able to contribute to the machine box code base i think would be more difficult than

other projects and so that's one of the reservations i have against open sourcing it is is that but yeah so it's really an audience question i think if people care deeply and know a lot about machine learning then maybe they're going to want to pick up TensorFlow and tackle it themselves. If you're an app developer and you want to quickly, you know, make your make your software smarter, slotting machine box in is just the quickest way to do that. Yeah. And I think it's like not inconsistent with other trends we're seeing like TensorFlow estimators and that sort of thing, right? Which is intending to kind of give these modules to people that will let them practically integrate things. Yeah, exactly. Yeah. It's kind of overlapping. They are catching up with machine box. And that was a great transition when you were talking about the tooling and under the hood, I assume you're talking about TensorFlow there.

What other tooling are you using? Where are you using Go, if any? Love to know how you guys are putting the pieces together. Yeah, so the basic stack is in Go. So we basically probably more than 80% of the code is Go because more than 80% of the code is just APIs and network calls and these kind of things. And the machine learning models, the training is done in Python and our favorite frameworks are Keras and TensorFlow. That's mostly what we use for deep learning. We use other ones,

like more traditional machine learning, things like ballpump-bubbit is a really old C library that I quite like. But basically, that's it. This is not so much machine learning code, we serve all the models in Go, um, many people in the AI space are doing Python, they're doing C++. You don't hear Go as often. So I'd love to know why that for your selection. Yeah, so Go has a deliberately limited language feature set. I once was speaking to a group and I said, you can't do that many things with Go. And it got a laugh

because I realized how it sounds. But what I meant was the actual language itself doesn't have that many features, which forces the code to therefore be simpler. You know, in some of the more modern languages with OO, you have big type inheritance. You've got all these language features that allow you to build really quite complicated, very clever and complicated things. The Go philosophy is around simplicity, which mirrors exactly what we're doing at MachineBox. So it fits brilliantly. Essentially, all of our code is,

Go code all kind of looks the same. So it's all familiar and you get such kind of benefits at development time, but actually more as you maintain the project, you know. So that's why Go wins, I think, from our point of view. Plus, we're fanboys of Go. There's no denying that. We met at a Go conference, you know.

Pretty much, yeah. But also, some people are really surprised when they ask, and they may hear about Matchingbox in a blog post or at a conference they contact us and say oh yeah I like your product just out of curiosity how many people are you well it's just Matt and me

developing we have some business side with Aaron but it's just pretty much three people company right now and the people get quite surprised like oh you you did so much um you have so many boxes so many products in in like two people developing and one business developing yeah and the answer isn't that we're awesome although david is the answer is uh that we we we are very selective about where we what we do so we we

deliberately don't do as much as you could do there's loads of possible things we could push into facebox for example and some of them tell you where the eyebrows are i haven't yet seen a good use case for why you need to know in an image where the eyebrows are but maybe there is one but until that you know until then we then, we're not going to invest all that time and effort and also add that kind of complexity to the API. So yeah, it's because we pick. We're very selective about what we do. We pick the things that we think are just the gold

from all this potential kind of complexity. And we just sort of focus around telling that story and solving that problem. So that's how we're able to do so much, it seems, I think. Go is the perfect tool for our philosophy. It feels really well into that mantra, into that mindset. So it's the perfect tool for us. I think both of you guys are awesome, just to set the record straight.

Thank you. I was fishing for that. That's why I said it. I'm glad you picked up on it. I figured you were. And not only that, but you've given me my next blog post idea, which is around eyebrow-based analysis.

Very important stuff yeah you can you can detect sarcasm with it that's the only use i think yeah someone's with you or maybe anger matt with you if you had that sarcasm detector wouldn't it be pegged most of the time yeah it would uh you can basically just return true okay that's a sure yeah that that would be 99.9 accuracy there was one time where i said something serious and wasn't being sarcastic but i forget what it was now so you you've talked a lot about kind of your your technology stack why why you've chosen go one thing i'm curious about mean, so I think everybody should use MachineBox in one way or another, but there's a lot of people out there maybe that are working

on data science teams or data engineering teams or whatever it is, and are, you know, maybe using TensorFlow to develop and train models that are getting deployed internally into their own sorts of services and products. I'm curious, because you consistently produce such high-value models that are integrated into your products, do you have any advice around that progression from training your model to getting it deployed within some type of service? Whether that be, you mentioned testing, you know, testing might look differently for machine learning models or AI models than in other cases. But do you have any kind of advice and insights around that process from, you know, training your model

to actually integrating it into a service, whether that's integrating machine box into your service, or maybe that's integrating your own model into your own internal service. Yeah, so I don't really know. So most of the problems are just technology that usually technology, you just get it solved one way or another. So there are a lot of tools coming up these days

that solve that problem, well, including machine box, but also in TensorFlow, the deployment is getting better so but i think most important is people so how this machine learning thing is transforming the way that people see software especially talking with customers now we have well you know in machine learning we have a lot of false positives, false negatives. Once you have something in production, they come up with questions. Sometimes the question that most of the customers ask, oh, we have this problem.

Well, that's not actually a problem. It's just a false positive. And there are ways to deal with false positives and false negatives. And changing the mindset to accept that a thing is not a bug. It's a false positive in a machine learning model. It changes the way that you interact with people. It's like, oh, you're not going to have a machine learning that is 100% accurate. So you have to deal with these situations. And that situation is just the way that we are mostly struggling or just trying to get the right conversations with people.

And I think that is going to come up in any software development in the next couple of years. Yeah, our job, one of our big challenges is communicating what's actually going on like you know we thought we're just going to deliver face recognition apis that's it or image recognition image classification or personalization apis and we found that quite quickly we did we did actually have to get into the conversation a bit more about, look, we don't expect this to get everything right 100% of the time. We expect it to do a much better job automatically than you're doing. Hopefully you can get it to the point where, you know, the exceptions that you have to deal with, if there are any in the workflow, get smaller and smaller. But yeah, that's definitely been

something we've had to focus on is is communicating that this is a kind of unlike other software where you do something and you get a result you don't like that's a bug and we've had some bugs opened where it says i put this image in and it didn't find the face you know and of course the image the face is like turned to the side or it's got a weird shadow on it or just something is weird about it and then we kind of get into that conversation it's well it isn't really a bug i mean uh you know it's kind of part of the expected workflow the question is how do we then tackle that going forward from a data scientist's point of view someone did

actually ask if they could put their models into our boxes because they knew the the building the models bit they were good at that but they had no idea about getting things into production and running them at scale one of the things one of the very early kind of rules that we gave ourselves and this comes this is kind of common sense now i think a little bit but comes from david's experience building at massive scale for the olympics in particular was that we had you know we we had to be able to horizontally scale the boxes just by adding more of them you know because scale is you know it's fine if you get this awesome technology and it works nice and slow on one machine but to to to really get the value from

it in most cases you want to run this thing at scale so that it can really you know get through work that it needs to get through and so we did we spent a lot of time also which you don't really see apart from the fact that it just works but we spent a lot of time in making sure that this these boxes could horizontally scale in a kind of Kubernetes environment where it was just elastic up and down as you needed. And of course, you have to think about what's the state inside the box. How does that work? And various other sort of, you know, will just load balancing across the boxes be enough to get what you want? Or is there more that we need to do?

And where does that happen? and all those kinds of things. So, yeah, it's been a great, it's been a great sort of experience building it. And it's even, it's more fun when people start integrating it and paying for it. That's, that's when you really feel like you've created something valuable. Yeah, that's, that's great. And, um, I can definitely resonate with, with some of the things you said around kind of exceptions in models and that sort of thing. I think people too often, in my personal opinion, you know, think about an end to end machine learning or AI model that does everything all the time correctly. And I think that's, you know, to some degree, the wrong thought in a lot of cases, because, you know, when machine learning models fail, it's the same, you know, we, we have an opportunity to refactor them, right. Which is, is in the end,

a good thing, right. So just to kind of, you know, getting, getting close to the end here, I was wondering, you know, again, what, what you guys are doing is kind of setting some, some standards as far as interacting with machine learning models. And so I'd love to get more advice from you guys in terms of the skills that data engineers or just kind of developers who don't really consider themselves data scientists or AI researchers, what sort of skills would you kind of recommend them looking into or what kind of skills do they need to begin to start integrating machine learning into their into their applications? I think that I don't think you need that many depends how deep you want to go into it. The trajectory that I would recommend

to somebody who didn't have any kind of idea about it would be to start by consuming APIs. And because if that's good enough, if that works for your case, then you don't have to do anything more. And that's what we've found so far. A lot of our customers have said, you know, we're just going to kind of try this because then we can build MVP quickly and then later we might change it. And then that later never happens because, you know because the boxes are doing just such a good job that they don't need to then change it. So definitely like any kind of API skill around consuming APIs,

most people already have those already. And then I think beyond that, it's really just a question of, I think, understanding a little bit more about just the kind of high level concepts i would say would be useful like you know with with with classification box with classification box you can create your own classifier with with training a training set now with classification box you do need a good amount of examples for each class. So, you know, when some people start using it, they have just a couple of images, a couple of examples, and you can't really get a model that's useful from that. So learning things like the

sort of softer skills around machine learning, I guess, which is, you know, the kinds of data, the kinds of problems that machine learning is good at, first of all, then what kind of training data are you going to have? Because machine learning is only as bad as its training data. So I think those sorts of things would be useful for everyone to have. And then if you're getting into more machine learning technical stuff, then I don't know. Yeah. So in my opinion, you should focus in one type of problem. So the machine learning is quite broad.

So if you want to get started, there are many different subfields. So probably just focus in a problem that you have or you want to solve, like, I don't know, sentiment analysis or classifying text or something more or less straightforward or in machine learning world, more or less easy and learning by doing it instead of focusing in maths or things like that. You can get easily lost in that sense. So try to solve, try to learn by doing. Solve a problem that you have and see how it goes. Once you have that working, you have that boost of energy,

just have something that is more or less working. Maybe it's not the state of the art, it's not very accurate, but it's better than random. So the machine is actually learning, and that is a good feeling. And probably just that is good to get started and get more curiosity and learn more things. That sounds great. So let me ask one last question for you as we wind up. So many of the listeners that we have are trying to figure out how to get into machine learning themselves.

And they might be software developers. They might be business people who are intrigued by what's possible here. on making AI technologies available and recognizing there's so many people that may want to either supplement their own business that they have or create a new business. What advice do you have for other entrepreneurs that might be interested in taking the same adventure that you guys are now a couple of years down? What would you say to them? Yeah, I would always say, solve a specific problem make sure you're solving a real problem this goes for any kind of software actually but it's too especially machine learning because it's all cool and sexy and and hard like machine learning is hard so

anyone that like david said if you make some ground, you get really kind of big rewards for doing that, like just emotional rewards you get. So yeah, it's kind of difficult to make sure that you're building something that has some true value. You know, because if you're just building cool tech, then there's no guarantee that's ever going to be anything. And often you end up building something that technically is brilliant, but actually doesn't quite fit the problem. And then you have to basically move or change what you're doing so that it does solve a real problem. And that can be quite a painful transition.

Usually involves adding loads of complexity because you weren't quite, you weren't really thinking about those things from the beginning. So of course you want to be able to evolve and learn and move, you know, a project along. But I would say start with a real problem that you understand. And the problem shouldn't be anything to do with machine learning, but machine learning might be part of the solution. Great. Yeah, that's wonderful advice. And we'll include links, of course, to MachineBox and other

things that we've talked about, you know, TensorFlow and Keras and Docker and Kubernetes. If you're not familiar with those technologies, we'll include some good links to getting started with those and learning more. And just want to want to thank David and Matt one more time for joining us. It's been been great to have you here and really excited about what's going on with MachineBox. Thank you very much. Yeah. And good luck with the podcast. I think it's awesome. I can't wait for future episodes. I'm sorry to everyone who had to listen to our voices for this episode, but future ones I'm sure will be even more interesting. Yeah.

Thank you very much. Thank you. Thank you. Appreciate it very much. All right, everyone. Thank you so much for listening. You can find more episodes of Practical AI at changelog.com slash practical AI.

Subscribe to the show and leave it a rating or review in iTunes. A special thanks to our sponsors, Airbrake, Linode, and GoCD. Our bandwidth is provided by Fastly. Learn more about them at Fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at Rollbar.com. And we're hosted on Linode servers.

Head to Linode.com slash Changelog. The music is by Breakmaster Cylinder. This show was mixed by myself, Tim Smith. Thanks for tuning in. See you next week.