Drill to Detail - Drill to Detail Ep.22 'SnapLogic's Enterprise Integration Cloud' With Special Guest Craig Stewart

Episode Date: March 20, 2017

Mark Rittman is joined by Craig Stewart to talk about application and data integration, ODI and Sunopsis, SnapLogic's approach to hybrid on-premise/cloud integration and the rise of data preparation and dataflow-based cloud integration tools.

Hello and welcome to another episode of Drill to Detail and I'm your host Mark Rittman. My guest this week is another fellow at RIT and someone I got to meet around 10 years ago actually through his role as product manager for Oracle Data Integrator and now he's at SnapLogic, a cloud data integration vendor based out of the US. So Craig welcome and thanks for coming on the show and why don't you introduce yourself to the listeners. Thanks Mark so I'm Craig Stewart and today I'm going to be a vice president of product management at snap logic um so that's what that's what i do at snap logic um the the the americans might not quite get the uh the the uh the history of that one but uh i'm uh at snap logic um a cloud integration vendor enterprise cloud integration vendor um based out of san mateo

in california uh we've we we go back some time back to the days at oracle i've been with snap logic now for just over six years it was my six year anniversary uh this week and actually a lot of people um just pinging me this week because of the notification on LinkedIn, which has been lots of names and faces from the past. Okay, so we met probably about sort of 10 years ago now, and you came into the world of what I was working in. So I was working as a consultant in the partner space for Oracle, working with tools like warehouse builder um

tools like uh you know obviously oracle data integration and so on and you were a pm then is that right for the product was it a pm or were you more in kind of you're in the field weren't you really um i was i was i was um not actually a pm um but uh i i was always a kind of a bit of an oddity in the Oracle structure because I actually remained as a part of the overlay for Europe. As we, when Synopsys got acquired by Oracle, we became an overlay. That first year, I actually found out from the head of SEs in the UK, so the sales consultants in the UK, had this interesting conversation where he said, oh, we ought to get together. And it's like, oh, sure. He's the head of SEs in the UK, fine go and talk to him and we sat down and

we were chatting for the best part of an hour and then he looked across the table and he says you don't know do you what's that then martin it's like you're in my organization from the end of the year like oh really oh how about that was this this Martin Cooper? Was this Martin Cooper? Yeah, that's right. I was quite well with Martin. It was just that funny conversation. It was like, he's looking across the table at me.

How can I say? I had some intuition about what was going on, but I hadn't actually been told at that point. Excellent. So, I mean, the way i got to know you first of all was through odi and odi when we're going to talk in this in this podcast we're going to talk about kind of etl going forward nita and the cloud and the work you're doing at snap logic and generally sort of directions on there but i think it'd be kind of interesting just to

kind of i suppose set some context really and talk about a bit of your history really to i suppose in a way to kind of give people a flavor of your experience in this area and how that kind of informs you know what you're talking about so so you you were working with me and some of my colleagues on on ODI and ODI was one of Oracle's kind of uh you know attempts or or product areas really around kind of data integration yeah what what got you interested in ODI at the time and what was what was interesting about that product and the market i suppose at that point you know about say sort of five ten years ago yeah so i actually joined oracle through the acquisition of synopsis and the i had joined synopsis through um an old colleague who I'd worked with previously who was running marketing for them. And I had been in the data application and integration space up to that point.

And when the opportunity came up to join them, it was a different implementation. It was something different the way they worked because it was ELT rather than the ETL. And I'd been involved with the things like Essential and Informatica over the years. But the approach that Synopsys took and ultimately ODI was this, you use the database as the transformation engine, which made an awful lot of sense. And so that's what was different. And then in the context of when we got acquired by Oracle, it was interesting because we were competing with the OWB guys.

And now looking back on it, you can see oracle do this with with many of the tools you you you um you you acquire things so that you're competing against yourself and now whichever one of them wins you win so so you know that approach that uh that oracle had you know meant that either way they were going to win and so so, you know, and as it was interesting that the ODI technology, which was strong on being database agnostic and working with the different databases, but actually having a, I suppose suppose a little bit of a clunky interface in comparison to uh the efforts that had gone into owb and to see them actually then really combine some of that technology going forward was uh was very interesting because you know that was part

of the weakness uh of the of the odi technology it was interesting because i think that if you look at so just if anybody that if anybody doesn't know the arcane differences about various oracle etl products from sort of 10 years ago and so on um you know it was an interesting time because warehouse builder was a tool that it's a classic situation where everybody hated it until it was taken away and and everybody used to moan about warehouse builder and i think it had its kind of quirks and that's a kind of a charitable way of putting it but it it was a in database etl kind of tool that would that would generate you know pl sequel which is oracle's kind of procedural language for sort of sequel

it would generate this and and it was it was popular i would argue because it was free and so an oracle i think we're in a situation where um they had a tool that was popular, I would argue, because it was free. And so, and Oracle, I think, were in a situation where they had a tool that was popular with developers, but they couldn't monetize it. And there was also that strategic kind of issue where it really wasn't designed from the outset to be kind of, I suppose, you know, just as, it wasn't designed to do ETL on, say, SQL Server or on DB2 or on Teradata. And I think when Oracle went down the route of making all the acquisitions around things like PeopleSoft and so on, they needed to have an ETL tool that would work on different platforms. But what was also interesting was that ODI wasn't that different to sort of OWB. And it must have been interesting within Oracle to come along there with a product that in some ways, you know, you were in line with Oracle's thinking around sort of using in database ETL but you know almost sometimes that they wouldn't say the hatred but the often in in in life you know when there's very few differences between people actually that kind of in a way causes the most kind of friction really it must have been interesting to come in there and and I think the outcome has been good

but what was it like coming into Oracle at that point, really, with a product that was different but not that different to OWB? It was challenging. And there were the meetings with some of the product managers on the OWB side. And I don't think there was any love lost between the two groups we say um yes but having said that that was you know professional um you know uh uh yeah conflict but actually you know some of those people now that i'm actually over in california um i'm i have met with them uh socially and you know it's like we we we look back and uh and laugh on those days and it was yes it was it was um uh i think there were merits on both sides and you know but it was it was a good good competitive situation yes i always heard that

one of the weird things about odi was the i think it was a lot of the concepts were translated from french weren't they so there were a lot of words and a lot of kind of concepts in the products like flows and things that just didn't make any sense to anybody that had english as a first language i mean is that is that an urban myth or was that kind of true that that was the case that was absolutely the case that you know that it was uh we kept finding uh spelling mistakes and things in the in the ui and the documentation for many years yes yes and there was one funny story i remember you came and spoke at when we'll get on to kind of snap logic in the future in a second but yeah you came and spoke at an event of ours uh which is bi forum in brighton and it was the first one we ever did i think it was that kind of strange kind of penthouse thing on top of a top of a hotel in london in brighton with that

burnt out pier in the in the background which was always funny but there was that seven i remember even now the release number seven nine five two release of odi which was i mean tell us tell us the story about that really that was that was that i mean that that was an interesting version, wasn't it? The 7952 was the packaging of ODI for OBIE. Yes, and the BI apps. And the BI apps. And the team down in Bristol, I can't remember who that was led by um had had spent an awful lot of effort you know translating informatica workflows into into odi so that they could ship an all oracle product and you know it was uh they stretched the um the capability shall we say it was interesting wasn't it because it was a one

off release wasn't it as well i think now that's what the bi apps runs its etl on but this was about five years before it kind of actually got done properly wasn't it that's right that's right but uh you know that that team down in bristol spent so much time trying to make all that stuff work to have a single consistent oracle product and the the configuration quirks and things around it were quite something but you know it i think it actually drove odi forwards quite a lot it did it did and it's one of those things where you see a lot of the a lot of the features that were then were then built into odi over time you could see were being piloted really then. I mean, I think the funny story that relates to me with that was that I think I spent about six months of my life learning that release and kind of working out how it worked.

And then I remember you sort of saying to me at the time, just be aware, this might just be the one release of it really. And it was, I mean, there's obviously a story behind that and the reason you had one version and so on but that was when we first met and and it was you know ever since then we stayed in contact and uh and then you ended up and obviously after a while you ended up leaving Oracle and you went to to SnapLogic so tell us a bit about the story of I suppose what interested you about SnapLogic and what led you into this world of kind of cloud etl and cloud data integration well in those last few months that i was with oracle um i had a number of situations where um we were needing to interface with some of the cloud applications that were emerging at the time you know obviously salesforce Obviously, Salesforce being one of them. And when Synopsys got acquired by Oracle, we had just migrated to Salesforce. And on the day of acquisition, basically, our Salesforce access got cut off because if you're on the Oracle network, you can't go to Salesforce. And it was, you know, the whole cloud thing at the time

was a, I don't know if it was right to say it was frowned upon within Oracle, but it was, you know, certainly not important. And even one of the customers I was working with was trying to use Siebel on demand, hosted Siebel. And essentially, I had to turn around to the customer and say, actually, you need to go and use something like Informatica because Oracle don't have a tool to integrate to that. That suits what you actually want to do there. because Oracle don't have a tool to integrate to that, that, you know,

it suits what you actually want to do there. You could, you could integrate with that using SowerSuite, but SowerSuite is, is a whole complex beast. And, and, you know,

doing integration to an application like that really shouldn't need to be that complicated um and so i had a number of those incidents come up and then towards the end of 2010 um i was kind of thinking well you know there were changes that were happening and so i actually got approached by a recruiter for snap logic and it sounded interesting because it was cloudy you know this was an integration tool it was cloudy and looking at the people that were behind it, it was people like Gaurav Dhillon who had actually founded Informatica in the early 90s and run that company. And they actually tried to recruit me as the first SE in Europe when Informatica actually set up in Europe. And that was going back to, what, 96 or something. I was at Sybase at the time and actually having quite a lot of fun doing an evangelist role there.

So they pursued me quite hard at that point, but I turned them down. But I got to know some of the people. And as it turned out, so I said, well, you know, this sounds like it's an interesting opportunity. So I went and interviewed with them, and I spoke to somebody on the phone. Then they said, okay, well, our VP of Europe is going to be in London, so why don't you go meet him? And I looked at the name, and I thought, I'm pretty certain that's the guy

that I'd got interviewed by before and you know walked into the reception of the uh the mario of marvel arch and it's like yeah we've we've done this before we've been through this whole interview process and it's like oh well in that case um well let's go and have a bottle of wine and find out what each other's been doing for the past 15 years and it was it was it was very much like that and we really hit it off very well and at the time they wanted somebody to run the technical side of the operation in europe so it it all fitted all the check boxes i wanted to do something different the cloud was emerging and the technologies that they were talking about integrating were you know not only databases but these emerging applications the likes of source and net suite and those kind of things

and you know it just really gelled as a as a as a concept uh you know for for you know it was in at that moment it was the right thing yeah and i think to be honest you know you were probably a little bit ahead of your time there as well because i remember when you went to snap logic um yeah i'd heard of them but i think that the idea of doing etl or data integration or application integration in the cloud was kind of you know it was it was it was kind of there but it wasn't as big a deal as it is now and i think that's purely that's probably more because you know we're catching up over here and and maybe the oracle world is is less focused on the cloud but certainly i think it seems like a prescient move that you made at the time and

you've been there ever since really yeah so so you know i having joined there the company at the time was focused more on the mid-market um and it was a commercialization of the open source technology which they had developed and so at the time that made sense it was really running as a hosted application for each customer and it was at that time that snap logic actually took on funding from andreessen horowitz and said well we need to turn this into a true cloud product and they built from scratch a true sas based product so where it's an application that runs in the cloud that all users are using the same version there's no version control issues all of those things um and so built a completely new product on you know with with that in mind. And that was built sort of 2011, 2012, and released in early 13. And by that time, I had been running the technical side of Europe,

ended up running all of our field technical people and you know that actually meant that i was the only field technical person outside of north america and i was running the team so every friday night much like this one actually i was on calls until 10 o'clock at night talking to my team and all of those things and so it was at that point that gora who runs the company then turned around and said well you know why don't you come and work out of headquarters and manage this product for us and my kids had both gone off to university at that point and it was like well we're empty nesters why not and a couple of years

in california i can cope with that so craig i mean you're i guess you're from scotland with a great name of craig craig stewart i mean how how did you make the decision to go to california i mean that that must have been quite a tough one really to go from uh to go from scotland in the winter to california well i'm i'm not um haven't been resident in scotland but you know st albans to uh to san francisco is is a bit of a change absolutely um you know it i've nearly moved to the states with work three or four times um uh you know with the um cybase and power soft and various other opportunities that we've had and so it's quite funny the kids my got two kids that were away at university and they kind of went yeah mom and dad have talked about going to you know a number of times it'll never happen you

know and it was when we uh you know rented out house, moved into a flat and said, we've got our visas. We'll see you when you want to come and visit us. Fantastic. Fantastic. Fantastic. So tell us a bit. I mean, this is a good opportunity now, I think, to tell us a bit about what is SnapLogic, the product? Paint a picture of what it is, really, what problem it solves.

And we'll drill into a bit of the technical bits in a second but but tell us a broad sense you know why would somebody be interested in in snap logic you know and who's it for and so on so the the product we have today we call the enterprise integration cloud so it's a sas integration platform so that anybody that wants to do integration in the enterprise today and by integration we're talking about data or application integration so data integration by that we mean mean more of the database type integrations. But equally today, that has gone and includes the Hadoop type technologies, as well as the cloud data warehousing, the things like Redshift and Snowflake, Azure SQL Data Warehouse, those things, which we see has having really taken over from the old query offloading. Oh, instantiate a MySQL on the side

and query the hell out of that rather than impacting production systems. Well, now today we see major customers actually taking the option to use things like Redshift where you pay for what you use as the technology of choice. So we do that integration, and it's very much a drag-and-drop and configure interface. And as the product manager of this product, one of the things that I'm adamant about is I don't want my users to have to go in and write any code so that they should be able to achieve their integration tasks by dragging and dropping and configuring now now that is quite a stiff challenge our developers have to build the snaps the the the connectors to these different applications and technologies to abstract the complexity of the APIs so that it is possible to have not only the flexibility

of and the ease of use, but also to be able to actually twiddle the knobs that need to be twiddled when you want to do more complex things. So it's very much about giving customers the ability to integrate applications and technologies and databases, etc. databases etc uh in a an easy to use interface which is browser based so there's no install required so you're not using an eclipse based user interface or anything like that this is you go to elastic.snaplogic.com in your browser and you use that to build, manage, and monitor all of the integrations. And we separate what we call the control plane, which is that user interface, the management console, and all of those things, as well as the application that's behind that, that stores the metadata, that does the authentication, that monitors the infrastructure and everything else. We separated that from the execution plane.

And the execution plane, what we call a Snaplex, can actually run. We can provide it as a service in cloud or what we see in most enterprises is that what we call the data gravity means that that needs to run on-premise today because there are few customers and few enterprises that are all cloud based you need to be able to reach the oracle database on premise or the the the sap system on premise and integrate that with some of these cloud applications whether it be redshift or whether it be salesforce you can run those things behind the firewall and have access to all of those things. Whereas if you use it purely in the cloud,

you have that barrier of, well, the firewall administrator knows how to say no. What don't you understand about no? Can I reach into those systems from the internet? So what we do is we run what we call the groundplex on-premise. It only makes bound connections. So that enables it to actually run on-premise. It's self-maintaining. And that's one of the important things about how we built this. The infrastructure itself, once it's installed, once that Snapplex is instantiated on-premise, all of those, the nodes that make that up, are automatically going to be kept up to date. or whether it be updates to the connectors to different applications,

we deliver those automatically into the application, and it's self-maintaining. So the customer can concentrate on the business of their business rather than the business and complexity of keeping up to date with all of those things in the applications, everything that they're using. So a vendor like, say, Informatica, for example, would also have on the surface a similar story. So it would say that we can run ETL in the cloud, on-premise and so on. What's the kind of the differentiator? What's the kind of innovation that SnapLogic would have that would be different to that and would be, in your view, a better solution for kind of hybrid data integration?

So our focus has been enterprise integration and you can do those integrations with something like PowerCenter. PowerCenter isn't ideal to be run in the cloud. You've got the complexity of maintaining the database that holds the metadata and all of those things. With SnapLogic, you just use the integration functionality that we provide, and you have just the execution engine on-premise. So firstly, in comparison to the traditional ETL tools, this is a true SaaS product and maintainability of that. Secondly, those companies like that, Informatica, have a cloud offering, but it was built for simply integrating endpoints like Salesforce and not dealing with the complexity of both cloud and on-premise in a single unified user interface. So with SnapLogic

you've got this unified user interface which does the functions of both what used to be called ETL and EAI, so the application integration. And things like we deal with data as a document, not as rows and columns. And you've dealt with XML, and dealing with XMLml odi had a relatively good solution for xml which was the the driver for xml which basically shredded your xml into a pseudo relational structure but you then had to deal with each of the tables that made that up with snap logic we deal with the document and so if the application you're dealing with gives you an XML document or a JSON document, that's natively going to work as that complete document. We don't have to break it up or shred it to actually be able to use the data inside it. okay okay so tell us about what so you mentioned snaps and snap plexes and so on so so that that

again sounds like a sort of a differentiator a key part of the architecture so tell us what a snap is and tell us about this a bit more about snap plexes and so on so so so the snaps are um are the intelligent connectors to the different applications so when when So we've got something like 400-odd snaps today that address different applications, enterprise-type applications, cloud applications, and technologies. So that ranges from things right the way through from the databases, the SQL servers, Oracle's, Postgres, MySQL, etc.

So the database type technologies, the cloud enterprise cloud applications, a lot of things goingSoft customers that are up for migrating. And we see a lot of activity in that space, actually, as well as the Salesforce and the ServiceNow and those other cloud applications. Oracle have been out there buying lots of them, the eloquence and those kind of things um and you know the snaps basically abstract the complexity of dealing with those things so yes you could use something like a a soap api and you know the one one of the the things that we see ourselves we see competing with other products in the market is you know the the tools that have a soap api and you you can you can uh join these things together it's real straightforward because it's got an api you know this whole api economy thing yes but somebody then ends up having to write the

code to do that integration um with the complexity of something like SowerSuite. And so what our objective is, is to make that really simple and abstract. So when dealing with an application like Salesforce, for instance, the Snap handles all of the authentication. It handles the keeping, do any do any keep alive and things like that that need to be done as well as you know if you're retrieving a set of data automatically doing the iteration of the result set so that you you keep that streaming in and that's just a drag and drop artifact that you drop onto the the canvas so one artifact rather than

having to define all of that uh all of those different actions they're encapsulated and obviously it you know um does the introspection on the application so it will then suggest you know if you're dealing with a database it will will suggest, you know, these are the schemas that you have access to. These are the tables. Also, basically dealing with those kind of things. And one of the other things is that we do schema on read. So if somebody goes and adds new columns to the database, it doesn't break the integration. If there's new things that appear in the XML, it doesn't break the integration. Right there's new things that appear in the XML,

it doesn't break the integration. Right. So we had this actually interesting. I was going to bring this into the discussion. We had stream sets on the show quite a while ago, actually. And it's kind of interesting. Again, it's kind of similar but different to you guys. So they talk about this this this this kind of uh issue about schema schema evolution over time and they talked about how they handle um how they handle kind of schema changes

and new columns coming in and so on so and and and you know they had various kind of in-memory structures and so on there for handling that how does snap logic handle schema evolution then and and i suppose schema changes over time then what you mentioned it handles it works with schema on read but what's the sort of technical solution behind that and the idea behind it so so the idea is that you know the the applications today aren't as static as as you know what that which we had got used to. And you look at these applications like NetSuite and Salesforce, and the schema changes with each version of the API.

And for customers, trying to maintain as they go forward those things becomes a nightmare. So that's why this idea of schema on read of, okay, well, we're going to accept what you give us. It's going to have the various fields. And like eBusiness Suite, with eBusiness Suite, there was a fixed way that you get additional

columns that relate to each each different object well in the new world that's very much more flexible and you can just add those fields and you know it's like the no sql databases it doesn't matter you can just have what what actually comes so snap logic will take what comes and you you're only fixing the detail if you refer to a field um then then it's going to um you know actually look for that field that's the only things that are going to be fixed in the the data flow um of course actually even that doesn't need to be need to cause any problems because what it's built for is built to be flexible and so that even if one of those fields is not in the the schema set that you're presented with, then it's going to go, okay, well, I'll just be lenient and I won't do that mapping if it's asking for that field.

And so the whole idea is that you have less friction when you're building these integration flows. Okay, okay. So I remember we spoke you and i spoke a couple years ago when i think when you when you uh when you'd moved to uh to stream set sorry to uh to to uh to snap logic and and one of the things that you were talking about a lot of the time was application into application integration versus kind of data integration and you showed me at the time demos of connecting uh this sas app

to that sas app and working at a higher level of abstraction than just working at table level and so on so is that still a kind of part of what snap logic doing is that again a differentiator for for your product absolutely so you know the being able to flow data between different applications and to have, in fact, what we've increasingly been doing is pre-building those integrations. There is only so much abstraction that you can do. So if you're flowing orders from opportunities that have closed in sales force and flowing them to something like workday yeah workday has a fairly rigid structure shall we say um and so what we can do is actually pre-build those and then they can just be offered to customers okay you need the integration that does that flow well here's the pre-built one and because of the flexibility that we have in terms in the schema on read type

thing that allows us to have those pre-built patterns that then only require a small amount of tweaking to address the specifics of that particular customer. So those are quite important. So those are the patterns in the SnapLogic user interface. And there's a building library of those. So not only have we got the snaps that address the applications and technologies, the specific ones, but there are these pre-built patterns that are pre-built sets of snaps that flow together to the different endpoints and the like. Okay. Okay.

So let's kind of move on a little bit, really, to sort of thinking about, I suppose, where the industry's going now and so on and a couple of things that that I've noticed in the current kind of role that I'm doing is a lot of talk around data integration is around data flow and and kind of pipelines and things happening in real time and things like Apache Beam and and so on really and also bizarrely you know but but but this is the case a lot of stuff now is hand-coded so if you're working with apache beam for example or cloud data flow on on on uh on on the google platform it's more common that you would use it would code stuff so let's cover though interesting i want to cover both those things there but let's start with pipelines and and data flows so what's the snap logic view on on data pipelines and data flow kind of ETL and that sort of thing?

So I try not to differentiate where you have a flow of data. In SnapLogic terms, that's a pipeline. And that pipeline in itself can actually trigger other pipelines and so it can be what more of what we would call a process flow so it a process flow is just a data flow that causes pipelines to be executed and And that could be scripted. And what we actually see is people using various different ways that they want to integrate some of these applications. They want the integration to happen either on demand. So I want to schedule the execution of this pipeline at a particular time.

And I might have an enterprise scheduler like Control-M or Oracle's enterprise scheduler that can trigger them. They also might want them to be invoked on a change in an application. So using an outbound trigger from those applications so both pretty much all of the cloud applications can trigger events much like database triggers where they say well when this event happens make this rest request to snap logic and every action within snap logic every pipeline can be exposed as a rest api that's out of the box that you get that functionality that then allows you to integrate snap logic and integration isn't always the center of a customer's world.

Usually, the center of the customer's world is the main application that they happen to be dealing with at this point in time. And so integration sometimes might be regarded somewhat as a necessary evil, shall we say. I don't like to think of it that way. I think it's essential. Definitely not on the evil side. But being able to be integrated rather than's from a command line or from another application or an enterprise

scheduler of some kind. That's a pattern that we see our customers implementing all the time. And being able to be triggered with those real-time flows and we have the concept of triggered pipelines, which are the ones that you make a REST request and it may be moving one row of data. It may be invoking a whole other process. So we have those triggered pipelines and we also have this thing which we call ultra pipelines. And, you know, that kind of the implication there being there is much lower latency on those requests that go to ultra pipelines because the pipeline itself is already running and we just marshal the data in and out of that automatically for the customer

without them having to build the infrastructure of putting in place a you know an http server and or loading up an app server or something that's all provided as part of the product and so you can then use these ultra pipelines and we're talking you know overhead of milliseconds to flow data through an ultra pipeline because the pipelines all already running and it can do any sort of request response type operation can actually be done using that mechanism and with millisecond type overhead and we've got a number of customers including some banks interestingly enough in the u.s using this technology to service their customers one of the interesting things there is that that that ultra pipeline

capability you know the cloud is ultra reliable right yeah except for when s3 goes down i was yes yes that was uh famously yeah that's right yeah um and you know theic itself, a lot of the network and can continue processing even when those cloud-side resources become unavailable. That's an interesting one for those kind of banking-type customers and those that have to have four nines and five nines. Technology enables customers to be able to do that and at a fraction of the cost of the HA of years gone by. Okay, okay. So you're doing a role, a product management role at the moment.

And one of the things, one of the parallels that I've spotted, I think I've seen really in the data integration market that is similar to BI is how you've had you know classic kind of massive each enterprise each data integration companies and products and so on but you've also had this kind of thing where where you've now had the rise of sort of data prep tools so so data I mean Google announced their data prep tool last week and that's the analogy for me is is in kind of desktop BI tools where end users can start to kind of do little bits of data integration themselves. And also you've had this thing where a lot of stuff now is coded by engineers as well. And so you've had quite a lot of kind of, I suppose, kind of changes in the data integration market that kind of made – I just wondered, again, let's take a start on that. So first of all, coding, you know, one of the things that is really strange these days,

but is the case is a lot of ETL now, a lot of data integration is hand-coded. And you might see, so if you worked with Apache Beam, it's hand-coded. If you work with things like Morphline, it's hand-coded. What's your view on that really? I mean, is that legit or is that just kind of like a bit of a regression or what really um one of the reasons why people are hand coding i think is that these technologies are you know haven't matured enough for graphical user interfaces to be built on top of them and and one of the the things that we actually built was what we call snap produce which was using that that same graphical user interface

with drag-and-drop objects and automatically generating the map produce code in the background who to build those those integration tasks in MapReduce. And we then implemented the same thing in Spark. Yes, SparkReduce? No, it's not Spark. We definitely didn't have quite the same ring of a name to give that one. But that graphical user interface on top of these technologies is something that as the market matures people because you know i'm i'm in i live in uh you know san mateo in california and that area in the valley the resources to find someone to work big data you're competing with

the Facebook's and the Google's paying extraordinary salaries shall we say and most companies simply can't afford those kind of resources to do data integration or data preparation tasks, data engineering tasks perhaps. And so the evolution from people writing code that the Googles and the Facebooks can afford to have, this is the versus the normal enterprise companies, the quality of, I'm not sure what the phrase is. It's inconsistent, isn't it? Yeah. The level of capability that you have to have

to use Apache Beam and these other things, level of capability that you have to have to use Apache Beam and these other things, the Kafka and the like, that barrier has to be lowered for them to gain mainstream adoption. And it's only through

putting a graphical user interface on top of them that they are going to get more widespread adoption. And otherwise, what you're going to end up with is yet another one of these really promising projects that nobody actually had people that were clever enough to implement for long enough and that they could keep hold of for long enough before something else came along to to drag them away whether that be another technology

or somebody paying more money for it okay so what about what about data prep so data prep is is is the other kind of i suppose another kind of development this area where uh you know it's where putting putting kind of more basic user-driven data preparation tools into people's hands has been again another area I think Tableau for example built some stuff into that some of their BI tools you've got Oracle's own offering in this area Google have just done something I think with Trifecta and so on but what's your thoughts on what's your thoughts on the data prep market and and the use case that's meeting and so on? What do you think on that?

Let me ask, the question that we ask ourselves in that space is, is data prep a feature of these technologies or is it actually a separate product market? And given that the tools like Trifecta and those kind of tools are so narrow at this point, we see the likes of Tableau adding that kind of functionality. The likes of Burst are expanding their capability in that space. So all of the visualization vendors are also looking for that space. The database vendors also need to address that space. And frankly, I see that space as being squeezed from both sides and it's it's one of those areas where it's important because as more people start to do more with data and people are looking for more forward-looking analytics rather than rear-view mirror analytics they need more people to

be working with the data and making that easy is very important so all of the people with the data are going to be providing more of that and all the people with the visualization tools and the machine learning and those kind of things are going to be looking to make it as easy to enter that space as possible integration goes across all of these so in pax arta another example of a company i think that i mean i've often wondered with pax arta could be due to do they could do to the enterprise data integration renders what say tableau did to kind of oracle and and and so on there and then you've got things like amazon glue i mean what i mean so amazon glue is is a is a is a pre-announced i think sort of product from

from a charitable way of putting it a kind of pre-announced product from from amazon that appears to be the kind of holy grail really of of kind of data integration you know using things like classification using things like auto generation of etl i mean you know have you have you sort of had a look at amazon glue have you kind of got any views on that at all i mean it's obviously an interesting competition thing there with with running your stuff on amazon as well but what's your view on on on there being this kind of in a way holy grail silver bullet really for this kind of work it's it's um it's it's good healthy competition without a doubt um and you know the the as with all of these the the other areas amazon's glue is mostly focused on Amazon's own products. And there is movement in the world where you get the anything but Amazon in the same way that there was an anything but Microsoft or anything but Google.

So I think that anything that's within one ecosystem, much like Oracle have some good tools around the Oracle technologies, but they start to fall down when you are going wider than that set of technologies. So that's kind of where we see that at the moment the other thing with amazon glue is it's again still requires quite a degree a depth of knowledge yes of the technologies and uh the capability of the developers to use it. Yeah, exactly. There's a high barrier to entry. And to be fair, you know, it's not actually out yet. And so it's hard to kind of to react against something that, not react against it, but have an opinion on something that actually is more kind of like a couple of pages on the web.

But it looks, I mean, I think the interest and the innovation and the focus and the investment in the data integration market is always kind of interesting. So, you know, I welcome it in my respect. So I'm conscious of time now. But one thing I want to ask you was, so it sounds like you and SnapLogic have made some kind of good bets that have paid off in terms of kind of focusing on the cloud

to run things as a kind of multi-tenant SaaS service and so on there. But, you know, going forward, you know, what are the problems that still need to be solved you think for customers what are the things that what you know things like to put out there things like data governance and and and stuff like that i mean what what are the problems that still have yet to be solved that people like yourself are kind of investing in and thinking about and and focusing on really for the future uh Metadata and governance, without a doubt, is one thing being able to move data. But within an enterprise, there is still a plethora of different applications

and technologies that are being used. And if you look at the Cloudera environment, there's the Cloudera director, and that can show you what's being used where in that Hadoop infrastructure. But, of course, what about what happens upstream and what happens downstream? And so actually that's whether you regard it as a standalone metadata space or whether it should be a feature that everybody is done a rip and replace to use SnapLogic across the whole piece. Other customers, it's just not feasible to rip and replace the existing integrations that they've got. Talking to one customer last week, they've got something like 14,000 integrations.

And you simply can't rip and replace those things. That's an evolution. you know having that metadata layer that can tell you firstly where everything is flowing and doing the impact analysis the data lineage those things but also being able to get that beyond the uh the etl tool itself you know as we did with the the odi umDI in the 7952 thing, where you could start to get the metadata in the report that you were looking at that would tell you where that had actually come from. And that kind of thing is certainly what people are looking for. And with data governance requirements,

that we see as something that we need to figure out amongst ourselves in the industry how to deal with. And that's one of the things that certainly I and my team're looking at um or whether it's you know an extension of these these same things like like uh the the uh director at cloud era okay okay so so where would people just to finish off then where would people find out more about snap logic and and maybe sort of like ask for a trial or just get more information really where where give us a bit of a kind of guide to that really so yeah we're a cloud company everything's on the web um so www.snaplogic.com naturally and you know within that actually we've as it happens um just gone through a revamp of our website this very morning um so launching the enterprise integration cloud um branding and

things so we've got a whole load of new content and things up there um and you know that that's the the uh the first place obviously there's there's lots of stuff there um we've got bodies on the ground in here in the uk so we've got a team based in the city in the uk i think we've got uh 11 people now in the uk um and you know also in the us obviously um but also a team in uh in australia um serving australia and new zealand market we've got some some uh good customers that have uh have been gathered now outside of north america as well but you know we have those people and obviously you know that you can certainly contact them through the website and the usual info at snaplogic.com

um will will come through to our team and they will direct to wherever people need to. You'll also see us at things like the Gartner AADI conference, particularly in London in May, for instance, and various other events around the globe. Fantastic. Well, Craig, it's been a pleasure speaking to you again. It's been a few years, but certainly it's been, yeah, fantastic. You sound like you're doing a very interesting job and a very interesting company

and in a market that is still just as relevant and just as being invested in and important to customers, really. So thank you very much for coming on the show and have a great weekend. And, yes, great to speak to you again. My pleasure, Mark. And thanks for inviting me. Appreciate it. Cheers. Thanks. keep in touch bye