Drill to Detail - Drill to Detail Ep.34 'Qlik, MSAS and the Strategy of Data Analytics' with Special Guest Donald Farmer

Episode Date: July 12, 2017

Mark Rittman is joined by Donald Farmer to talk about his work at Microsoft on SQL Server Analysis Services and Integration Services, why he moved to Qlik and the challenges of evolving a BI product strategy from focusing on desktops to focusing on the enterprise, and some advice for customers, software vendors and partners working with data and analytics tools.

So my guest this week is someone many of you will know from his work at Microsoft as part of the SQL Server Analysis team and then Integration Services after that. And then more recently from his work at Qlik as their VP of Innovation and Design. So none other than Donald Farmer. So Donald welcome to the show and thanks for coming on and speaking to us. Oh thanks very much Mark it's a delight to be here. So I can obviously tell from your accent Donald that you're from Scotland so tell us a bit about I suppose your kind of background and how you came into this industry and how you ended up writing your first predictive application which I understood was for fish farming. That's right. Yeah, gosh.

So I started, I mean, I worked as a consultant, as a technology consultant for some years on my own in Scotland. I actually started as a historian and an archaeologist. And I was working a lot with computers at that time, building databases and data link systems. But then I actually had to make some money. So being a consultant was a good way computers at that time, building databases and analytics systems, but then actually had to make some money. So being a consultant was a good way of doing that. And from that, I got into kind of working in fields, as you say, like fish farming and hydroelectric, things like that, and creating systems for that. And from there, I got into working in building products,

not just sort of in consulting, but actually building products within a great little company in Aberdeen in Scotland called AppSmart. And we built products on a Microsoft stack, which were effectively data warehouse rapid development tools. I worked with a great team there, including some well-known Microsoft people now. Ewan Garden is that big figure at Microsoft. And then from there, I moved to Microsoft itself in 2001. Worked there for just about 10 years. Worked on analysis services and integration services, which was my great pleasure to work on that. That was a tremendous product, a tremendous team, and building something from scratch was very exciting. Worked on the data mining team, which was very cool, and started the Power Pivot project. That had really started as an incubation through the

work of Thierry Derr, who is now at Tableau working there but he really incubated the power pivot project i took it over to put it into production to actually make it a product which is kind of what i specialize in you know productizing things um and then having done that it was time to do something else so i went to click um helped to build their new product click sense and again that's a building a product from scratch almost, which was kind of exciting. And then having done that, and then as Qlik matured after about five years at Qlik, it was time to do something else again. And now I'm independent and I advise investors and product companies on data analytics, strategy, innovation, that

sort of thing. Fantastic. And I'd like to get onto that later on. I mean, I think, like you say, the advice for the kind of product companies, advice for investors, that's kind of very interesting. And I think your views on where the market's going and where the opportunities are will be really interesting. But to kind of set the scene, really, so you, I mean, I got to know you through your work on analysis services. And back in those days, I was working largely on Oracle technology. And we always saw analysis services and back in those days I was working largely on oracle technology and and we always saw analysis services as a fantastic product that was you know I personally found it very very hard to compete against when I was in the consulting world and it was just so well put together and

certainly comments I've heard is people describing the work you did as kind of built using love and so on I mean what do you think was what in in hindsight was particularly kind of good about analysis services and the work you did there? Yeah, what worked well there? I think three things came together. One was we had a really solid understanding of the business need, which came from the leaders of those teams. People like Bill Baker, who was very well-known, had come from Oracle. He had been at Oracle. He had been at IRI before.

And also people like Corey Salka, and we brought in people like Joy Mundy, who came from a really solid kind of data warehousing background and eventually went on to work at the Kimball Group. So we built a really sound business sense. And that was enhanced by building great customer relations. So we had a really good sense of the business problem. And then we had some pretty inspirational kind of technical people around.

We'd bought in, acquired a team, actually part of the Panorama company, and that team came with people like Amir Netz and Moshe Pasomansky, Sasha Berger, brilliant people, brilliant, brilliant technical people. And then on top of that, we had the, or maybe underneath it, I should say, we had the Microsoft infrastructure. And one of the things you learn when you're at Microsoft, one of the things I really learned, and I think everybody who's been at Microsoft on a product team and then moves on, takes this with them, is just how difficult it is to build solid enterprise software. And you get that infrastructure at Microsoft. You get that DNA of just how to constantly ship.

And it's difficult. It's difficult to ship enterprise software. And you needed that support. So many startups have great ideas, but they're not able to build that enterprise infrastructure. So those three things came together. Great understanding of the business problem, the technical genius, and the Microsoft infrastructure, which enabled us to build this really solid product. I think that's really the secret.

Yeah, definitely. I mean, so you then went on to work on integration services. And so that, again, was an interesting product in that it, to my mind, it took a slightly different approach to the kind of push-down ETL tools that I was working with. It was very kind of hub-and-spoke. It was very developer-focused. Again, what do you think worked well with that,

and why do you think that was so successful as a product? Again, it's a kind of mix of things. One of the most important things about integration services is the original mandate for building that product was to make sure that Microsoft stack was relevant on big what we called in those days big iron by big iron we meant you know machines which had wow a terabyte of storage you know blow your mind you know I think I've got a USB stick here which is about a terabyte but you know that

was the that was the idea you know We had to make Microsoft relevant in that space. And it's important to remember that Microsoft is all about Microsoft, it's all about Windows, it's all about the stack. And so integration services was necessary because if you're going to have a data warehouse that runs on a big iron, if you want to run the world's largest data warehouses,

you've got to be able to get the data in. And there's not just enough to put the data in, you have to be able to transform it. So you need an ETL tool. So the mandate came from that feeling that DTS as it was, data transformation services, wasn't that. Yeah, people love, and this is an important part of it, people love DTS. It was a great developer tool. It was really good for what you might now call data wrangling, just getting stuff together and working at work. But it wasn't an enterprise class ETL tool. It wasn't going to compete with Informatica or Ab initio. It wasn't going to load a terabyte of data in a world record time. So we had to build a product that would do that. But at the same time, we had

to keep that developer love that we had from DTS. And I think those two things, you know, coming together, those two mandates were kind of really interesting. And then, of course, you put together a great team of people. Again, it's so important putting together a good team of people. We had Kamal Hattie, he's a leader who now runs Power BI and was just the best manager I ever worked with. Mike Blazsai was a developer of Genius. Kirk Hazelden was a dev manager of Genius and went on to lead

Microsoft's master data management product. He's written a lot of books about integration services and things, great guy. And then of course, fantastic test team, which was super important. Testing was so important to make that work. And again, that comes down to the enterprise capabilities. So again, great team. And then those two mandates, super large scale developer, lovely. Yeah, I remember at the time being a consultant we're again working with oracle technology we used to use dts to move data between oracle databases

because it was so much easier to use than it was so much easier than anything else exactly yeah and and i was reading an article i think it was about you a while ago and talking about how you worked with the data mining technology in that kind of space as well and you put data mining technology into the kind of etl processes there is that something that you can talk about? I mean, that sounds quite interesting. Yeah. So Microsoft not only has development teams, we also have Microsoft Research, which was a tremendous resource of super smart people doing fundamental research. One of the things that was interesting was how do you get that technology into the products? And not many people were doing that. I mean, Microsoft Research was coming up with fundamental research, but sometimes it wasn't being productized. And not many people were doing that. I mean, Microsoft Research was coming up with fundamental research,

but sometimes it wasn't being productized. And we just loved working with the team. So one of the things we came up with, myself and one of the leaders of the data mining team, very smart guy called Xiaoyi Tang, and we developed a system which used data mining algorithms. Now, data mining algorithms look for patterns in data. So you look at existing data, find the patterns,

and then you can apply those patterns to new data. So, for example, you could look at data which is coming in, say, of customer data, and you can run a clustering model on it, and you can find clusters of likely customers. And that's a very classic data mining algorithm. A new customer comes along and you can say, which cluster do they fit in? But there's something else you can do, which is you can actually look at that and say,

here's a customer which doesn't fit into any cluster. Maybe there's actually something wrong with the data. You know, if this customer is an outlier, then there's actually something wrong with the data. If this customer is an outlier, then there's potentially something wrong. So what we did very uniquely was we took the ETL process and we integrated that kind of predictive analytics into ETL process. So you could run data through all the usual transformations of joining and merging and you know filtering and so on and then you could run it in the ETL process through a data mining algorithm and say does this does this data smell good doesn't does it look as if it's an outlier or not and if it's an outlier we

should probably put it down this channel and have a look at it later and make sure it's actually okay and that was was very adaptive. Nobody at that point had built that kind of adaptive intelligence into ETL. People are catching up with that now. I'm seeing a little bit more of that from companies like Alterix and Trifacta and Talent, but at the time it was very unique. Yeah, it's interesting. I mean, so what's your take on, I suppose, the data engineering movement, data wrangling and I suppose kind of, you know, vendors And the reason for that is simply because the use of data is changing so rapidly. It used to be when we built data warehouses,

when we built what SSIS, integration services, was for, the process was pretty straightforward. You had a business model in mind. And you took that business model, you created a logical model, physical model that represented your business. And then the data was transformed to fit that model because you already knew the kind of questions that were likely to be asked.

So you've structured your data in order to better answer those questions. But today, with big data, with schema on read, with artificial intelligence, with machine learning, with all the stuff that's going on on the data science side, you really don't know what your data is going to be used for. You don't know what somebody is going to ask tomorrow. So it's very difficult to follow that structure.

There are still parts of the business, the financial system, which is well-structured and has to have its data structured in a particular way. That's fine. Those data warehouses will probably exist for a very long time. But most of the time, you actually don't know. And yet analysis is data preparation. Data preparation is an essential part of analysis.

When you look at data, if you're a data science asking a question, it's still true that 80% of the work is getting your data in the right shape. So now you have to have a tool which is adaptable, responsive and flexible enough so that you can get the data into the shape you want without necessarily knowing in advance what that shape is going to be and certainly not knowing overnight what that shape is going to be, not be able to build a repeatable ETL process and that's where this adaptability comes in. So Analysis Services has gone through quite a few changes over the years, from the original multi-dimensional storage to the more recent tabular storage option. So what was your involvement in those changes? And looking back, what's your view on how they worked out? That was starting as I was leaving

Microsoft. I left Microsoft six, seven years ago now. So that process was starting. We were looking at the impact of in-memory processing in particular was clearly the huge breakthrough at that time. Power Pivot was our first product that did that. So that was starting. And certainly at the time I left, people were already starting to experiment with tabular models and ways in which these analytic problems could be answered in memory. But I wasn't part of that process, or at least I was only at the very beginning. It was certainly quite a controversial change at the time. Chris Webb, who came on the show earlier in the year, talked about the big impact it had on his world

and certainly the controversy and the divided opinion there was at the time about how wise a move it was to move from Multimensional to Tabula and MDX to DAX and so on. Well, you know, it's funny you mentioned Chris. Chris is always a tremendous part of our community and never backward in coming forward with suggestions and criticisms in the best possible way. I don't mean overly critical. I mean, he kept us honest. Yeah, exactly. And I remember when we launched PowerPivot and we included the DAX language in it. And DAX was an attempt to kind of take the complexities of MDX, the multidimensional query language,

and make it at least easy enough or at least structured in such a way that it could be used within Excel by people who understand Excel formulas. And I mean, as you know, there's now been a lot of books written about DAGs and a lot of work done on that. But at the time, that was our aim. It was to make this so simple that really you could learn it in the same way as you learn Excel formulas. And Chris and several others of the people in the community were, oh, you can't do this.

You know, this is our living. You can't make this so easy. It's our job to help people with this. These problems aren't easy. And so we created this very easy language. And I remember at the beginning of that launch that Chris, in some ways, was quite upset. You know, you're making this too easy and you're hiding the complexities.

And not only that, you're going to damage our business model and we're your partners. And I remember about six months later, he came back to another event and said, yeah, I'm quite happy. It's not that easy. Yeah, he sounded happy when I spoke to him a while ago but so yeah I mean things always change in business and IT and then

you moved on yourself to to click and I remember at the time I remember reading at the time it was a quite a momentous thing you moving on to click really I mean so so the role you were doing there you said you you look you launched their new product so click sense there so what was that about really and what was the challenge you faced there and what was the kind of the difference between that and their previous products right yeah you know i so the first thing i should say is that when i moved to click there was absolutely no negative side of that at all you know it's i i love my time at microsoft um and it's kind of funny the very fact that people saw it as a momentous thing was one of the reasons i left it was i'd become so built in

with the bricks almost at Microsoft that it was, you know, what do you do next? And so it was kind of exciting to move somewhere else and to see what they were doing. And what was interesting for me about Click is that Click had just been through an IPO. So I wasn't joining a startup hoping to get a big payout. They'd already had their IPO. What I was really interested in was here was this company from Sweden, which had been around for a long time. I mean, I've been around for almost 20 years when I joined them. They built this very unique technology. They'd had a lot of success with it. And now they're at the point where they have to really become an enterprise company. They've got their IPO, everybody's watching them, and they have to move, they have to step up.

And what I'd said earlier in this podcast about Microsoft's ability to actually build enterprise products, that DNA that you get of that which was super interesting. So I wasn't looking for a startup. I wasn't looking for a company which was already building enterprise software. I wanted that challenge of how do you take someone to the next level? And Qlik were absolutely ready for that phase shift. And what was interesting to me in the work that we did was that Qlik needed to do two different phase shifts

at the same time, which is super difficult. On the one hand, they had a product which had been very successful for many years on the Microsoft platform, on Windows, and it was stuck. And I'll say it that way. It was stuck in a paradigm, which was very much the kind of the dialog boxes and checkbox sort of style of configuration of the product. Thousands of options, it felt like. Quite a lot of complexity, but a lot of depth to that product.

But not suitable for the new world of tablets and self-service and business users genuinely kind of building their own stuff. So it needed a complete sort of rethink of this user experience and at the same time they also had to move into this enterprise class of software. So very unique challenge and absolutely the most exciting place I could possibly have gone at that time because of those two challenges. And that was and it didn't disappoint. I had a great time there. The teams were amazing. The leadership was amazing. And you know, in some ways I didn't do much there. The teams were very functional. They're really strong people. And so a lot of time, my job was just sort of explaining to people why this was important

and keeping the team sort of focused on the business reasons, the cultural reasons, and the technical reasons why this was important. And so long as you could keep that, if you like, that internal messaging clear, a lot else followed from that, you know. So that was, it was a great time and so moving on again more recently you left click and founded your own consultancy treehive consulting to provide strategic advice to customers vendors and investors working in the bi market i'd like to spend the rest of the episode talking about the kind of issues and strategic questions you

get asked about by these three audiences so let's start by thinking about the actual customers of the BI software that we work with each day. What are the key success factors, the determinants of project success for end users and customers starting a BI project? You know, the successful customers are the ones who have the right culture and the right team in place, first of all. And the reason for that is because you can have all the software in the world and you can have absolutely the best software, you can have chosen software, which you think is going to meet your needs. But if you don't have the culture in place, then you're not going to be able to take advantage of it. And so part of this culture is, are you really going to be driven by the decisions that you make, the data that you have?

Or are you really just, in many cases, you still see a lot of companies which they have reporting more or less as a tax rather than as something that's actually transforming a business. So, you know, are the companies that are successful, they're the ones who really want to understand their business and they've got that curiosity and they build that in to their culture. And as a result, they're able to give people some of the freedom to explore and analyze. And I think that's really, really important. Exploration, discovery, analysis, that's really what makes the difference.

If it's not that, then it's just regulatory reporting and management reporting. And everybody does that and who cares? Culture is really important. And then the choice of tools is really what are the tools that support your culture and what tools support your data infrastructure as it is. You already have a data infrastructure. Every company has a data infrastructure. I'm a one person contractor and I've got a data infrastructure. I've got an accounting system and I've got a data infrastructure. I've got an accounting system, and I've got a customer system.

And it might be very simple, but I've got data, and I need to use it. So people already have a data system. And the idea, and this was true, I think, perhaps back in the 80s and 90s, where somehow you kind of built a data system from scratch and built an entire infrastructure. That's kind of gone because our businesses, every business nowadays is just thoroughly native digital. And therefore, the data is already there. And so what system will work with your existing infrastructure with the least disruption,

but the maximum value from that? And then how does that work with your culture? And when I say work with your culture, I mean, you know, some companies I've visited, financial services companies, are very structured. And so they actually need systems which might be more like what you would get from, well, maybe Oracle or SAP, but increased nowadays from, you know, classic kind of enterprise applications, which are very structured, highly governed, with a lot of control in them and requiring a lot of IT work in order to build out the complexity. On the other hand, you may have a culture which is much more flexible, much more agile,

and where people are wanting to use tools like Tableau or Qlik or Perpia because they actually want to do their own work. If you give the self highly structured tools to a company which is actually trying to be agile and genuinely do self-service, they're going to be frustrated. So you need to have a set of tools which matches your company's cultural profile, if you like. And I think that's where success comes from.

Okay, so when I was talking to, again, talking to Chris in the previous episode, he was talking about how even now, these days, he finds the buyers of the software he uses, Microsoft software, are typically the IT department. But also we hear about sort of, I suppose, the budgets moving to the business now and moving away from IT. I mean, do you see that? What do you see happening? And what do you think the impact of things moving away from IT to the business will be? Is it kind of good?

Is it bad? I mean, you know, what's your view on that? Yeah. Well, I don't know if it's good or bad. I mean, to be honest, it is what it is. But I think there's three things happening, which are somewhat, they're all related, but they're somewhat different. Yes, most of the buyers are IT departments. But actually, how does the IT department get to the point of buying software? In other words, if you're going to do the enterprise deal, you're going to buy 10,000 desks of something, that's typically an IT department purchase, or 1,000 desks of something,

that's an IT department purchase. But that IT department isn't starting from scratch. Probably in that organization, there's some click users or some Tableau users or some people have downloaded Power BI or some people have been playing with Watson already. In other words, business users have probably already adopted some software before it gets to that big IT decision. It's not as if you go in and there's a blank slate and nobody's got anything, and now IT come in like a deus ex machina and deploy 1,000 desktops.

It doesn't happen that way. There's always something in place before that big IT sale. And that is the important distinction. The difference between the business user trying something out and the IT department making an enterprise purchase is the way that budget is structured. And the IT department budget will typically come out of capital expenditure and business users are just expensing and taking it out of operational accounts. And then

that brings us to the question of cloud because one of the big distinctions of cloud and subscription software is that it typically comes out of operational budgets because it's a recurring operational cost, rather than traditional licensed software, which tends to come out of capital budget. And so very often what you're seeing, it's almost like an accounting process that's going on here rather than anything to do with BI in itself. You know, it's where does the money come from? Which budget does it come from? And where do we want it to come from? Part of the problem here is that people, both in the past and now with cloud services,

typically dramatically underestimate the cost of their software. As a vendor, I can tell you, it's always frustrating for me that people, buyers so often focused in the past, and they do now, on license cost. How much is the software actually costing me in terms of how much I'm going to pay for the license. But actually, the overall cost of ownership of software has got much more to do with maintenance, training, all those things, rather than license cost. License cost maybe is less, 10% or 20%. But how can a customer estimate that and

differentiate that between different vendors i mean we all know in a way that there's a cost to that but how how can that be useful in a decision really do you think for them or actionable yeah that's that's a problem where i say that you have to choose tools which fit with your culture if you have a tool which doesn't fit your culture then those costs are going to be very much higher because you're you're constantly kind of working against it it's be very much higher because you're constantly working against it. You're trying to tack against a wind at that point. So I think one of the things you have to do is very carefully run proof of concepts and experiments within companies. the cloud. It's so much easier to deploy and to provision a genuine land and expand strategy, what vendors call land and expand, where you start with a fairly small

deployment and build up. Why not do that in the cloud? It's super easy to do compared to a traditional system or a traditional process where you'd have to make quite a large commitment up front before you can even deploy something. The result being that vendors would go through this whole process of evaluations and their checklists and stuff. But if you still haven't tried the software and actually deployed it and see how people get on with it, then it's really difficult. And I've seen a lot of companies out there who have acquired software from, you know, Microsoft, Oracle, Business Objects, Tableau, Cognos,

and it's ended up being Shelfware because they've done what they think is an initial evaluation. The buyers liked it. Very often, people in the IT department love this software. A number of times, I find people who, you know, they love Tableau because they're data guys and they love the visualization. They deploy it to a thousand desks and, you know, 990 people don't like it. It's not they don't like it. It's not their product and they're not excited by it. Or sometimes it happens the other way

around. Yeah, it does take time and people are tremendously excited by it. So that's why you've got to find this cultural fit and you've got to try it yeah do you i don't know do you have do you have much dealings with say startups i mean because there's obviously a whole ecosystem of bi tools for example like looker for example and beam and there's a whole bunch of them out there that are quite different i suppose to well they've got some similarities to the old school tools but they're different as well i mean do do do startups have a different way of buying BI and implementing BI in your mind, really?

There's a lot of companies who are in the BI space now. You know, you look at the Gartner Quadrant or you look at the Gartner Cool Vendors list as an example. There's just a ton of good stuff out there and things happening. And, of course, they all want to have their one particular twist. A lot of them have grown out of visualization projects, academic visualization projects. I think trying to follow the Tableau route because they started as an academic visualization project. So a lot of people are doing that. And then you have companies doing some really unique stuff. I think Looker is a great product.

Great, very developer-focused, great for embedding. You know, very powerful for that. Pretty agile. I quite like that. And then there's companies who have been around for a while who are actually doing interesting things, like Logi XML are doing well. And then, of course, you have consolidation.

So companies like that first have kind of been acquired and might know what's going to happen there. So I think there's a lot of interesting stuff out there. One of the things that's important is, you know, when I say find a cultural fit, typically, I think we're going to move away from the idea that one tool solves all your business problems. It's you know, oh, yeah, we're a Microsoft shop, from the idea that one tool solves all your business problems. It's, you know, oh, yeah, we're a Microsoft shop and it's all part of BI. I think you're much more likely to find that every company has a portfolio of tools, including probably some smaller specialized BI tools,

which are just super popular in particular departments. So a research department will have one set of tools. An R&D department might have one set of tools. A production department might have another. Supply chain may have some tools that it just loves. And these may all be different, and that's fine. Yeah, yeah. I mean, that's a nice lead-in to start to talk about things

from a perspective of vendors, really, as well. And, I mean, I had Timo Elliott on uh show a little while ago who oh yeah who obviously you know from sap business objects and so on and you know that's a classic i suppose vendor who has bought a lot of products has a complete suite of kind of bi products there as well but i would i would suspect is finding it hard to be relevant in today's kind of marketplace or at least as relevant as the business objects days same with oracle as well well in what do you think the challenges are for the mega vendors out there I mean in today's market is it is it the same as before or is it changed or what really well the challenge for a mega vendor is

that they're mega and that's that's a huge that's that's a huge problem well it comes with the huge customer base and it comes with the fact that people essentially run their business on your products. When I was at Microsoft, one of the entertaining conversations we used to have, just about the sheer scale of the number of customers we were dealing with.

If you're in Excel and you decide to change the visualizations in Excel, you're changing the visualizations for 500 million people, many of whom have spent a lot of time getting their visualizations just right using the existing tool. I remember, you know, things like ADO, ADO, just the data connectivity systems, you know. I remember having a conversation with Alyssa Henry, who later went on to Amazon and Square, about downloads. We had put a little tool up. I think we had 2,000 downloads on the first day.

We were awfully excited about it. She reminded us that when they would release a patch for ADO, it would get 2,000 downloads an hour. It's just a different scale. Now, if you're at Oracle or you're at BusinessObjects and you have built BI products into your business suites, you have

at SAP and Oracle, you have millions of customers, thousands of customers around the world who are running some of the world's largest businesses on your software. And you've built it in a way that this becomes mission critical to them. That then becomes really difficult to change. And changing that is like, you know, you're trying to refit the engine while the ship is sailing.

It's really tough. And it holds back innovation. You can innovate at the edges, but really difficult to innovate at the heart of that product. So that's the challenge for them. And frankly, I'll be really blunt, neither Oracle or SAP have been up to that challenge. No, no.

You can just tell that. Yeah. They've both done it. Yeah. Again, I mean, funny enough, I work under one of the guys who was in charge of Oracle development back in the Oracle days. I mean, fair enough, I work under one of the guys who was in charge of Oracle development back in the Oracle days.

I mean, neither of us are there now, but talking to him about how that worked, and one thing that was interesting as well from what Oracle did and what SAP are doing is trying to kind of embed analytics and now artificial intelligence into their applications. I mean, do you think that's the kind of the way that they should be going to differentiate themselves in, especially, I suppose, do you think that's the kind of the way that they should be going to differentiate themselves in, especially, I suppose, also Salesforce with Einstein as well? Is that a route they should take, do you think? I think it's a route they have to take.

I've been having a little Twitter discussion recently with Rita Salam and Cindy Housen of Gartner about the penetration of BI. And they've been saying that, you know, penetration of BI is still about 30%. And the argument that I was making with them was, well, that's not necessarily a bad thing. Maybe that's the right number. Maybe 30% is, you know, if you think of BI as primarily being used by decision, you know, to support decisions, there's still decision support.

After all these years, it's still all about decision support. Then how many people in an organization actually make decisions that require that kind of support? Maybe it is 30%. And Rita came back and made the great point that, well, actually, if it's embedded and this analytics are embedded in the operational applications that you can use, then in some businesses, it could be 100%. If you're in a call center, and your information about which call to

take next and how to answer that call and how long you should spend on it is actually coming from a BI system, why not 100% penetration, which is a good point. So I think when companies actually want to own the entire work stream of a company, and that's absolutely the ambition of companies like SAP and Oracle and IBM, or if you want to own the entire sales process of a company, if you're Salesforce or the entire marketing process,

if you're Marketo, then it makes absolute sense to build embedded intelligence into those systems. And this is where the intelligence side comes in. Are you just giving information, in which case it's embedded analytics, or are you trying to give insight? And if you're trying to give insight, then building an intelligence systems would be absolutely the way to go.

And I see in the world of startups, there are some general purpose AI startups. There's people building algorithms and people building engines. people who are building ai into supply chain management or people who are building ai into you know the packaging software that runs packaging systems or people who are building ai into labor contracting systems or people building ai into hospital or hotel management systems that starts to get really interesting but i guess i guess there's i guess that's not an easy thing to do i mean obviously there's been the press recently about ibm watson and there's always been the kind of feeling it's their products have kind of, I suppose, underwhelmed. I mean, you're going back to the days of when you used data mining before in ETL and going forward to now. Why do you think it's so hard to get the value out of AI? Or why do you think there's a perception that it's hard to do? Well, I think the perception it's hard to do is because it is hard to do.

You know, the idea that this can be made easy or generic is a little misleading. You know, so traditional data mining was pretty, in some ways it was pretty straightforward. It was technically difficult. This is important. It was technically difficult, but it was pretty straightforward conceptually. You find patterns and you match data against patterns. And those patterns give you insights. And so when you use a recommendation algorithm on Amazon, customers who bought this also bought that, you're getting some insight from patterns and it's pretty useful. There's a lot of work to do to make it happen, but I think it's easy to understand. AI is a little bit different because in AI, you're giving the applications of AI even the simplest applications

you start to realize that that itself is a huge challenge because I mean this has been understood in AI for many years they used to call it the frame problem what is it that's important? how do you know what's actually relevant? the self-driving car how does it know what's important? it can look at all this stuff in the street how does it know what's important? It can look at all

this stuff in the street, how does it know what's significant? Never mind, I mean, can I look at everything and process every single variable? Well, even what variables do you identify in the first place is a challenge. So it is difficult. That's why embedding it into well-defined business scenarios is actually very effective because in that case you reduce the scope and you enable the AI to do what it's really good at, which is discovering its own patterns and discovering its own solutions within those patterns and often being able to do it in ways that are surprising to human beings and that's where the great advantage is okay okay so so looking at i suppose the the i suppose commercial versus open source question something that's something that probably you and i've noticed over the years is how much open source has sort of come into the world that we

work in so hadoop obviously is there um you know linux is a kind of an operating system that's very very kind of popular now within bi though it still seems to be that that's a largely a commercial tools kind of market did you see that open source being relevant in the bi market in the future is it relevant now why why do you think it perhaps hasn't been so sort of prevalent as it is in the database on the os kind of market any thoughts on that at all? Yeah, absolutely. I think the main reason is, in fact, because the BI tools that we're very familiar with grew out of either commercial database applications, but they were also targeted at enterprise deployments. And enterprises were traditionally reluctant to bring on open source products. And then you started to see some companies really being smart around the edges of that.

I think Talent did a great job of building an open source data integration tool, which was tremendously competitive and powerful and suitable for enterprise. Jaspersoft did some of that around BI, maybe not quite so successfully, but they did build a very good application. But enterprises were reluctant to take it on. But now what we've seen is, I think, one very significant shift.

Well, it's two very significant shifts. Of course, the Apache projects, the Apache infrastructure is so compelling. And by its very nature, it's open source. It could only have grown as quickly and as effectively as it has done by being open source. And that has been very powerful. And then the other thing is in the academic world, R really took over as a language for analysis, especially for data mining and predictive analytics. And that has become so pervasive that companies like Statistica, Statssoft, SAS, who have specialized, SPSS, companies who have specialized for years in predictive analytics have had to adopt R as perhaps almost a primary language in some cases, even though they have their own stacks.

And that's because that's the language that people have been learning on. They've been learning at university. They come out. They're fluent in R. They know Python. And so those are the tools they expect to use. OK. OK. So last question from the kind of vendor side. I mean, so you must have nurtured and mentored lots of managers over your your time and managed some good ones, some bad ones or whatever. What, in your view, it makes a good product manager looking after a BI tool in this space? What are the kind of the, I suppose, the qualities, the techniques,

the approaches that you see that work well for product management in this kind of area? You know, that is a great question. And one of the first things is empathy for the customer or empathy with the customer, I should think. And that is the ability to actually really understand what a customer is trying to do. I would generally say, for example, when I was at Microsoft, where we had summer interns used to come along. You know, you could have summer interns in development and summer interns in test, and they would do great work, often did outstanding work. Summer interns in product management typically didn't work very well. These are people who are currently at university.

And part of the problem was that over the course of a summer internship, they didn't have enough time to develop empathy with a customer. But they also hadn't seen enough of the world. You know, they hadn't been in business. They hadn't been out there working on a day-to-day basis. And so I actually find that experience outside of the field that you have specialized in is actually quite important. And if you don't have that experience, how to develop that experience, how to listen to customers, there's a great book by the designer Indy Young called Practical Empathy. And I think that's a great summary of what we need in product management. We need practical empathy.

We need the ability to understand what customers actually want and often to dig behind that and understand what the needs are that are driving that. Not just understand what the customer is asking you. This is not just about, you know, I ask a customer, send a customer a questionnaire and that's all I need. You need to get behind what the customer is asking for. Why are they asking for it? What are the real business problems, the real personal problems, if you like, that might be behind? So that's important. But the second part of it is that you need to be able

to synthesize that into a technical answer to that problem. That's also difficult. That comes with experience. But that doesn't mean that you need to be a coder or you need to be a great developer. But it does mean that you have to be able to take often very abstract concepts that are coming from business scenarios and somehow make them into a product, make them into a productized answer. And the third thing is that you need to be persuasive, influential. You're not building the product. You're not writing the code. You know, when I was a consultant myself and I wrote my own code, I had an idea. I made it happen. When you're a product manager,

you have an idea. You've got to persuade someone else to make it happen. And you've got a very, often very smart, very intelligent, very insightful, sometimes very opinionated developer on the other side who says, well, I want to do it this way. I want to do it that way. How do you persuade them to do it the way that you think needs to be done? And you can't order them. You've got no power to do that.

So these things are really important. and that's you can't order them you know you've got no power to do that um so it's you know these things are really important the empathy with the customer the ability to technically synthesize that into into a concept into product and then the ability to communicate that and persuade an influence these are these are the skills and these are all very soft skills you know yeah i totally agree i mean that coming coming from coming from a consulting background and running my own business and being the CTO, for example, I was used to basically if I say something, it happens. But the thing that shocked me coming to product management is you have no actual direct reports. All you have is the ability to win the argument, to influence people.

And things happen with a product and momentum gets behind a product because of your ability to actually kind of put a good argument together and to show there's traction there and so on and that's that as you say that that ability to influence them and empathize and so on is is really important it is yeah and you know i when i went to click i had a lot of fun with this or fun and if you like that sort of thing um because the it's a swedish company and the development team was based in Sweden. And in Sweden, people make decisions in a very collegiate way with a strong emphasis on consensus. Everyone's got to agree before they can move forward. And I was, quite frankly, I was burning the house down.

I had all sorts of ideas that were shaking up their entire business model and their entire customer model. I had to use my powers of persuasion, my powers of influence far more than I'd ever used at Microsoft, which was it was it was a challenge, but it was enjoyable, too. Interesting. So just a couple of last questions, really. So the first one is, from a kind of M&A perspective and acquisitions and generally consolidation or movement in the market, a few years ago, there were a lot of big acquisitions. There was IBM buying Cognos. There was Oracle buying Hyperion and so on. Do you see that continuing or has the market kind of settled down now? Where do you see changes in that happening in the future, really?

What's the direction in that area? It's like a forest fire you know every few years the forest fires sweep through the pacific northwest and um what comes afterwards are the green shoots which start to appear again you know and the all that dead wood has been burned away and i see consolidation like that you know um consolidation happens because companies get to the point where they're close to this tipping point between being a startup and being an enterprise company. And then they get acquired into an enterprise infrastructure or they themselves grow and start to make acquisitions and they become this enterprise infrastructure. But that leaves space and it leaves space for the new quirky tool that solves a particular problem that an analyst needs. And so when you looked at tools like Tableau, for example, and I think Tableau is a great example of this, a great visualization tool that came out of an academic project,

and virtually every Tableau user I knew in say 10 years ago also had other tools. They already had corporate BI, they already had business objects deployed by their IT department, they already had Cognos deployed by their IT department, but they had a particular personal need for this new tool. And now Tableau has become an enterprise product and you can go to Tableau customers, and you'll find that, yeah, they've got Tableau, but there's these two, three other people who've got some new thing that does what Tableau doesn't. And that's getting exciting.

And then there's always new approaches. I think one of the exciting new approaches that I've seen recently has been, well, we've talked about open source, and I think that's one new approach. We've seen company like Domo, you know, Domo, the Josh James runs, and they've done some fascinating work by focusing not on the analyst, the consultant, the kind of person that you and I have worked with so much, but going straight to the C-suite and saying,

we're building a tool for the CXO, who doesn't have time to build all this stuff themselves but needs smart, adaptive insights. And that's a different approach, which is, you know, I don't think they're as successful as they would like to be, but I think it's a very unique approach. So, you know, the consolidation and acquisitions, they leave space for these new vendors to come in with a new approach.

And that's exciting. Yeah, exactly. And last question, really, this is one that's personal for me, really. How do you see the services market changing here? Because, again, with the move to cloud and the move to, I suppose, the kind of the business people owning the budgets and so on, it struck me there wasn't such a market these days for large-scale you know enterprise uh i suppose implementations of bi tools how do you see the services market changing in the kind of bi space you know is it going to go away is it going to change what do you think on that you know um the services market in the past has been very focused

on technical knowledge. You know, we're technical experts who can come in and do the stuff that you can't do. And I think that has to change. The best people in services have always had that technical knowledge, but they've had this, if you like, product manager-like ability to take a business problem and turn it into that, you know, to turn it into that technical solution. And I think there's going to be a greater emphasis on that.

The idea of a service consultant who's just really capable of doing the technical stuff is largely going to go away. I think there will always be people who've got some technical difficulties. And I know one company just now who are really struggling with a scalability issue, which comes from a design problem in their database. Absolutely, they need a technical consultant who can come in and sort that for them. But how much of that work is out there? Not as much as they used to be because systems are becoming much more self-maintaining and self-tuning so so those problems are starting to go away but I think this ability to take a

business problem and sort of crystallize it into into a solution is something which many people struggle with and and that's where the the services industry is going to be successful in the future okay and and so that leads in quite nicely to what you're doing now so you obviously left click and you're now kind of have your own consultancy so what's the kind of the problem you're solving for people and and where what made you go into that kind of area really I'm really interested in the the the challenge of where data and analytics fit into company strategies. And my belief is very simple, that every company nowadays, every business is a data business.

We all have data. Every business has data, and therefore they need a strategy. Because if every company is a data business, then every company is an analytics business to get the best out of that. And if every company is an analytics business, then every company is an analytics business to get the best out of that. And if every company is an analytics business, then every company should be an advanced analytics business because we can use advanced analytics to get even more value out of it. So how do you, you know, particularly I'm interested in working with software vendors, what are you doing as a software vendor with the data that you're generating,

with the data that you're acquiring? with the data that you're acquiring? Are you getting the best out of that for yourself? Are you getting the best out of that for your customer? And how are you analyzing and adding value to that data? And so what I do, my job these days is I go to investors and I help them with who should you be looking at and who are the interesting data companies out there. But more importantly, I help them with their existing portfolio of companies. And they could be doing all sorts of different things, software for the legal business, software for the hotel business, that sort of thing. But what's the data strategy those companies have? And can we help them build extra value into their products and therefore get extra value out of their products by developing a data strategy.

So I work with investors, I work with their portfolios, and then increasingly I go out with customers as well and help them with their strategy. And then the other side of this is I have a kind of innovation and design practice, which is I like to go into a series of workshops with companies and really help them develop innovation strategy. I think that's a lot of fun doing that. Fantastic. And how will people find out about what you do? Is there a website you've got or how will people find out how to contact you?

Yeah, they can find me at treehivestrategy.com. Okay. We call it Tree Hive because we have this crazy tree house shaped like a beehive that I use as my thinking space. So treehivestrategy.com is my location.

Fantastic. Well, it's been fantastic speaking to you, Donald. I mean, it's really great to hear about the old days of kind of Microsoft and Click, but particularly, I suppose, now about data strategy, understanding kind of how the market's going and so on.

So it's been really good to speak to you, and thank you very much for coming on the show. Great fun, Mark. Thank you very much. It's great. Thank you.