Drill to Detail - Drill to Detail Ep.46 'Market Trends and Findings from the BI Survey 17' With Special Guest Dr. Carsten Bange

Episode Date: December 19, 2017

Mark Rittman is joined in this episode of Drill to Detail by Dr. Carsten Bange from BARC to talk about findings from the recently completed BI Survey 17 including the continuing move to modern BI platforms and self-service desktop tools, analytics adoption trends and the increasing incorporation of BI functionality within business applications, the surprising topicality of master data management and data governance ... and whatever happened to Nigel Pendse and his legendary OLAP Report?

Welcome to Jewel to Detail and I'm your host Mark Rittman. Today I'm pleased to be joined by Dr. Carsten Banger, one of Europe's leading market analysts for analytics and data management, and the founder of BARC, the people behind the BI survey that you might have seen myself or others in the industry mention and ask people to help with their field work. So welcome to the show Carsten and why don't you introduce yourself properly and tell us about Bark and the BI survey. Sure, thank you Mark. Yeah, I founded Bark about two years ago and our primary mission is to help companies to make technology and software selection decisions. So that's how we started. We started with a test lab and published basically research on feature comparisons and on successful architectures and so on. So that was back in the 90s when we called BI MIS. And pretty soon we added the user research to that.

So with acquiring the OLAP report in 2006, we did a big step into tapping into more than 2,000 users that each year answered about 50 questions around their project. So how did they select their BI product? What experience did they make? How do they rate the vendor and the implementation experience and so on? So that's the second field of research for us next to these more technical evaluations. We do now a lot of user research.

This is the BI survey. We also do the same thing on the planning market, so planning and budgeting solutions. And then we do a lot of topical surveys each year, looking at trending topics and how users feel about them. Okay, so how does the BI survey differ then to perhaps ones that are more well-known, the Gartner reports, the Forrester reports and so on? How is it different and how is the research different really in that? Sure.

Yeah, I mean, first of all, it's the pure size. So it's close to 3,000 users each year i think the second important point is the um the the data points we have have back into the history so we run it since the year 2000 so we have now 17 years of data and to see some trends emerging or going back is also, I think, quite interesting. And another thing is it's a really unbiased view. So it's not sponsored by anyone. So we ask the users about their experience. And it, I think, gives a very good view on what's really happening in the marketplace.

Okay. So you mentioned the OLAP report there. And I remember the days of Nigel Pensy in the OLAP report and he's fairly you know he's fairly opinionated and he was fairly the reviews he did at the time were very in-depth so what's the link back then to the OLAP report you say that you acquired the I suppose the kind of the content of that a while ago but what's the link back there really it was really um actually we started as a company out of a university project that was comparing uh software products so we we ran a test lab and we were going quite into detail and people really liked that and that was not

superficial but we really tested the solutions and we often checked more than a hundred features and and really the architecture of the solutions and when i looked around in the marketplace in europe i found only two analyst companies or analysts that had a quite a similar approach to ours and and one was nigel pensy in the uk and the other was actually the cXP group in France and so it was basically no didn't come to a big surprise that we teamed up with Nigel we had some collaborative work and at some point since he was looking to go into retirement at some point we then came to an agreement that we continue his work and after a while we changed the name from OLAP report to BI. So the OLAP survey to the BI survey and the OLAP report we called the BI verdict back

then and we tried basically to continue the work since we also had a very thorough methodology in testing products I think was a very good fit and we could continue that work. Okay. So, I mean, looking at your website and some of the things you do now, you've gone well beyond, I think, what was around then. I mean, the content you've got and the reports you do, a lot more than I suppose those days really. But it's nice to sort of pedigree there. And as you say, having the data going back all those years is interesting as well.

And we can talk a bit later on about trends that you're sort of seeing in the market but how does so how does it I mean I suppose a direct question how does it pay for itself for you I mean you're doing a lot of research there's a lot of time goes into that you don't take vendor support how do you pay for this and what's the economics behind it yeah actually times changed in that in the early days when we started users were willing to pay for research. So we basically sold our reports and we sold the product comparison reports and we sold the survey. We still do that today. But honestly, as we all see in the media business, things change quite a lot and users expect a lot of research to be free

these days. So what we do is that we basically offer vendors to buy some of the results documents and with that we make sure that we can finance the effort. But it's very important to see that this is only done afterwards. So we always run each year, we run the survey, we run the data, we produce the report, and then we offer it to vendors. And some obviously like what they see if they have happy customers and other vendors don't. And then obviously, they don't buy the marketing rights. Honestly, a lot vendors buy by still buy the report to see how they stack up and and to see the data but um this is basically how it's done interesting i mean i work in currently work in product management in a slightly slightly related sort of area and i know that certainly

we look at things like trust radius reports and things like that i mean even if the data is is negative you want to know that you want to know in areas that you're lagging behind. So you say you might not use it for marketing purposes, but you certainly would value it as an independent source of market knowledge really. So yeah, very interesting. No, absolutely. And what's the voice of the customer.

And I think vendors are very well advised to really take a look what data we can collect there. Okay, so you've just finished the fieldwork for the BI survey 17, is that correct? So tell us a little bit about, I suppose, the work that goes into that and the scope of it and then we'll get into the details of it in a second. Absolutely. Yeah, I mean, as usual, we have a very big sample size. So again, we are close to 3,000 participants. It's a truly global survey. So we have people from more than 90 countries participating. And I think what's also special about it, it has a very strong European

footprint. I mean, in our industry, you see a lot of surveys, but most of them are almost purely based on North American samples. And as we all know, things might be a bit different. We have different vendors and products in the survey, which are predominantly present in Europe. And often users in Europe also have a different viewpoint. Sometimes they are a little bit later to the market, which can be good and bad. And we see also that they sometimes take a different approach to BI projects. And I think that's quite interesting. And that's what we can see here, that we have quite a well-balanced global sample.

Okay. And I guess you're looking also at implementations on existing software, whereas I would imagine some of the surveys that go on other people, they're more interested in what is driving your license business, whereas you're asking people what are they doing actually now with software that's there as opposed to what are they buying? a broad distribution so for each product we look at we have some people on older versions and some on the newer ones so it's pretty typically a pretty good mix and that also shows then the the differences so how things might be improving or not but but you're right it's a it's a well balanced view on the products as well okay so i mean just getting on to the results of the last survey then so what do you what are you seeing in terms of the trends of new implementations? You know, is self-service really dominant? Is this idea of a modern BI platform, is it actually reality?

Is that happening at the moment? Is that how you're seeing implementations happening? Yeah, absolutely. I mean, this is not a new trend, obviously. We see that now for many years. But it's still the predominant way to implement BI today. So, I mean, we have seen that also the older, the larger vendors have reacted. I mean, the product portfolios are now quite different. But what, let's say, started with Qlik maybe about 10 years ago

and has been continued with Tableau, let's say, in the last 5 to 10 years, has really changed the market, I think. So it's really now about self-service. It's about the Qlik implementation. It's about enabling a business user to do a lot of things themselves. And what we see changes now

in maybe two interesting ways. One is that the maturity, obviously, has increased over the time. And we see now a much higher importance of the data management related topics. So it's not, yeah, absolutely. So it's not so much about, okay, let's implement a self-service tool everywhere across the organization in a completely ungoverned way. And everyone can do whatever they like. But the more mature organizations have seen the downside of this,

which is really basically in the consistency of data. And it's also in the repeating efforts that you have everywhere. So things are done repeatedly. And obviously this is not efficient to do. So we see the pendulum in a this is not efficient to do. So we see the pendulum, in a way, is swinging back right now. So we see more efforts now on data governance. We see more importance now in ideas or thoughts about how to actually control self-service, how to come to still a set set of defined kpis for example how to ensure security how to

ensure data privacy especially right now in the gdpr context which is also an eye-opener where you where you see okay if you have completely uncontrolled distributed data across the enterprise in hundreds of self-service installations, this will basically not help us or will not be compliant to what's demanded in GDPR. So it's really, we see the pendulum swinging back. It's now, again, discussions around platform features, security, privacy, consistency in data, data governance, maybe as the superset, the term that maybe is describing that quite well. But on the other hand, the users still demand the flexibility.

They still demand the user-friendliness. And for many organizations, we see it's it's quite a balancing act to to basically try to do both at the same time so is this is this new newfound discovery i suppose of data governance and and and those kind of things is that coming from the business users themselves who are rediscovering this or is it more that the kind of the it department are reasserting their authority or is it I mean how do you see how that's happening or you know is it from business or is it from IT coming really? It's now I think that's the good news it's now also coming from the business side I mean they

have in the in the self-service BI field they surely have taken control. So they own the budgets now. They are making the purchasing decisions. They are often even implementing the tools without IT support. But after a while, many users also see now the downsides of this. And basically, I think IT is much better back in the game now, being able to have this more centralized approach and being able to support the users here. And I think that's the important notion is that, let's say, five to 10 years ago, we had this discussion. IT is setting the standards and every business users that wouldn't comply to the standard, what basically was bad. They weren't supported.

They were basically uncontrolled. And then they started to help themselves. And now this has changed. I think the IT departments see that they needed to change their role. And many now take a much more supportive role. They understood, okay, it's important for the users to have that flexibility. And we have to define together with the users where are the areas where we need a central governance and where we can play a role. And, for example, where it absolutely makes sense to provide centrally governed data.

And where do we need the flexibility for the users where we are not trying to control it from from an it point of view they basically let the users have the the self-service and i think that's the the position we are in right now okay so do you find are you hearing that projects are more successful now with this approach i mean is there any you know with this move to self-service and the business owning the budget, has that led to more success and better ROI in your opinion? Yeah, I think yes. It's really that we see that speed and flexibility are two of the major requirements today. So, and this has in served quite well with the safe service tools and um we discussed the downsides of this but this is now um being better and better addressed

so um i think yes it's it's um has been more successful and users are often quite pleased with what they get there. Okay, so have you found that, I mean, the vendors like say Oracle, for example, the one that I know have made quite an effort to try and create what they might call kind of bimodal or kind of hybrid platforms where they try and cover both self-service and governed. Has that been successful? Have you seen that being taken up by the market or is that more wishful thinking do you think let's say it's starting slowly um so right now the um let's say pure self-service tools are still much more popular and they are actually adding these platform features um for sure they're not not there yet it's not comparable but um i think there's still a um there's still a

notion that the the pure play let's call them pure self-service tools are still more popular but as i said the maturity is rising and and more and more organizations understand the benefits also of having this bimodal approach okay i mean i know this might be outside the scope of your survey but do you also hear that there are data warehousing projects going on i mean is is that kind of thing happening or is it largely reporting against data in in situ now or doing blending directly in the tool when you know is is that a dead area or is it still kind of live really data warehousing? It's still alive. It's simply, people said it's dead and it's too slow and it's not too costly and so on. But after a few years

of trying everything decentralized and distributed, they see, oh, there is some value, but obviously we need to do things differently so um what we see now is that there's still a new data warehouse being built because it's basically a very strong concept it's very strong idea to have centrally governed data and you still need it but i think the the one thing is that the scope is now um better defined so it's more clearly defined okay what makes sense to have in a data warehouse. And the second thing is that we see that now, for example, agile development methods have been implemented, that there is tooling now available around data warehouse automation,

which also eases things and also speeds up, for example, the introduction of new data sources or changes in the data model and so on. So we have seen some advances that really help to make the data warehouse more agile. And I think a lot of companies have learned their lessons. But on the other hand, they also see the value. So I think it's not dead. We see the data warehouse. We see that a lot of data lakes have been built up, often for the more unstructured data.

But this now really plays a role maybe as a raw data storage, but still for the structured data, basically you cannot really get around the concept of the data warehouse. No, exactly. So what about cloud? I think I noticed in one of your reports that cloud adoption was perhaps not as prevalent as we might have thought a few years ago. I mean, is cloud deployment of BI now a reality or is it more for the future? It's starting for sure. Let's see, in our data we see quite clearly that the adoption curve in Europe in general is about two to three years behind North America, which I think is quite a big gap.

Actually, Europe is not homogeneous here. We see some regions being much quicker, like the UK, like the Nordics for example they they are were jumping on this topic quite early but what we also see is now that the later moving markets like France or Germany now the cloud adoption is really starting and so I think it's quite interesting and for me that will be the next big change in the market that we see this in a broader scale adoption in the next years. So is cloud when you when you see BI deployments moving to the cloud is that driven is that driven by other movements the company's

making into the cloud or you finding that BI is moving separately into that I mean how dependent is it on the whole company moving to cloud do you think it for sure helps we see now many companies having a cloud first strategy for example so they really look hard whether they can do new projects in the cloud what i found in find interesting is that typically there's no migration going on. So whatever you have today on-premise stays there. So it's really more about new projects or new data sources to look at. And with more and more data sources in the cloud, obviously now Cloud BI gets also more interesting. And the second thing I see is that there's quite a difference between the data management in the cloud and the BI in the cloud. So in for example cloud-based data warehouses I don't see a lot right now while it's easier to have a cloud-based

or software as a service BI solution running. So I also see a difference there. Okay okay so when when when customers move to the cloud, do they typically go with their existing vendor? Or does that prompt them to look at new vendors and new ways of doing things? I think it's a chance for new vendors, for sure. And we see that everywhere. So we see that on all levels of the architectures because data integration into a cloud solution is often done differently than on-premises. So there's a chance for new solutions and new vendors to take a look. We see that the cloud infrastructure providers like Amazon or Microsoft or Google are playing a much

bigger role whenever it comes to the data management side of things and we even see for example now first companies rethinking their Hadoop strategies whether they might be able to replace that with the native object stores in the cloud platforms. And now suddenly Amazon S3 or Azure or Google play a role where you traditionally had either the database players or the Hadoop distributors. So for sure the native cloud companies, let's call them that way, are playing a much bigger role now. Yeah, I mean, that's been my experience the last 12 months.

I used to be working with Oracle quite a bit, working with, I suppose, Cloudera, working with on-premise Hadoop. And now suddenly it's all BigQuery and so on. And the ease of use is fantastic. And it's probably quite worrying for the likes of Cloudera, for example, who had quite a big play in that area. But, I mean, do you see the Hadoop vendors still relevant in this space, really? Are they being referenced as much? Are they being used as much? Or is that kind of a moment that's passed?

Right now, I think they're still quite relevant because, I mean, a lot of data lake projects are still in the build-up phase. And I think especially for this data lake concept, for unstructured data, it will not go away, at least not quickly. On the other hand, we do see these vendors already starting to reposition themselves. So if you, just an example, I recently saw in London a keynote from one of the senior executives of Cloudera, and I think he didn't mention Hadoop once. So it was really all about the advanced analytics,

data science support. It was about having the metadata and the semantics layer across several stores and so on and so on. So I think that's a sign that the vendors are already starting to reposition themselves. Okay. So in general, usage of BI within companies, has it been rising? Has it plateaued? What's the general penetration like in terms of BI tools in companies?

And looking back at your years of data, has that kind of changed or whatever? What's the general kind of, I suppose, penetration like in terms of BI tools in companies? And looking back at your years of data, has that kind of changed or whatever? What's your view on that? Yeah, I mean, that's one of the interesting results of the BI survey that we asked the participants how many employees in the company are actually using BI tools. Not from one vendor, but across the board. And then we compare that to the size of the company. And what we see here is that the proportion, the amount of the employees that are using BI tools is rising, but very, very slowly so despite these self-service efforts despite bi tools getting cheaper and cheaper it's quite interesting to see that the notion of pervasive bi is not really taking place it's not really happening we if we take the average we are now at about 13% of employees in a company having access or using a BI tool. utilization, the notion that data and information is getting more and more important for companies

and decision making, then on the other hand, we cannot really see that in the data. And I think that's really interesting to see. Is that because you're counting, I suppose, usage of discrete standalone BI tools and people these days are using more BI embedded in the application or the workflow. I mean, is that where it's going or what do you think? Yeah, absolutely. That's, I think, one of the major explanation patterns here is that the BI capabilities or the BI functionality is rising everywhere and it's increasingly available also in the operational or transactional applications. And for example, with the large vendors,

let's take SAP as an example, we have really seen that the move has started now with the increased BI capabilities in the new ERP release S4. Many companies now have revisited the data they have in their BW data warehouse systems and also the approach they take to BI. And they have redefined that they see that there's quite a lot of operational BI going on. So BI amused by people in an operational process,

typically with data from only one source, the operational system, and typically used for quite operational decisions. So no tactical, no long-term, no integration of data from different sources required. And out of pure lack of functionality in the operational systems, this has been done in data warehouses and BI system, but now it's moving back to the operational systems.

And I think that's a good sign what's happening, that the increasing capabilities are being used and employed and that basically BI is getting really embedded everywhere. And I think that's one of the reasons why the BI tool usage is not rising so rapidly. So do you see, taking things kind of several stages further and talking about machine learning and AI, do you see that having an impact on things to the point where machine learning and AI is going to start to reduce the need for BI tools because the decisions are made for you? I mean, is that becoming part of the landscape you see or is that a long way off from now, really? It's starting to happen, but right now only in very distinct use cases. So whenever you have too much data for a human being to really oversee or maybe quite complex decision models,

or you need to take decisions in a very rapid manner, then we see that models are already doing that today. But that's quite a small area so i think right now many companies can still improve in increasing the the capability to have access to data to provide insight to their decision makers and that's still a major area of improvement for many companies but then slowly we But then, slowly, we see the move into more models being integrated into processes, and with that, decision-making also being automated. But for me, that's really just starting in industries like, let's say, online retail. It's quite normal to have that. But in many other cases, it's still quite far out.

Okay. So taking a step back to, I suppose, more traditional BI projects, if you look back at projects from the traditional vendors that would have a large governed semantic model with an enterprise kind of platform, are you finding that those systems are still being used for new BI projects? Are they getting that kind of reuse that we thought they would? Or are they all just being ditched for new tools whenever a new project comes along? What's the reusability like of these systems? It's there, but for some years it wasn't.

And that's, I think, the big problem. So I think the larger vendors that provide the platforms only caught up recently in the last years to provide something to the users that is really making sense for them. But for many years, the large platform or enterprise BI vendors did not really offer that. So what happened in most companies we see today that have a lot of BI projects and a large architecture and many BI implementations, that now we have a big coexistence. So we still have the BI platforms in use

often for standard reporting or older ad hoc reporting environments and then a lot of visual analysis has been introduced a lot of self-service and this is now living side by side since um basically the users didn't really find what they were looking for with the um with the enterprise bi platforms okay okay. And just as an aside, you mentioned Europe and the US being different in terms of usage and vendors and maybe even the UK and Europe as well. I mean, so within Europe itself, is it vendors such as SAP Business Objects, are they still big there?

Are there different vendors that are more popular in Europe than, say, the US at the moment? I think for the larger vendors it's quite similar. Obviously, we have some regional specifics like SAB being much stronger. I mean, it's their home market, especially in the German-speaking countries, for example. But apart from that, we might have a bit of a slower adoption since most vendors obviously start in the US. So, for example, Tableau is a good example. I mean, they started in the US, became very, very popular, and just then moved over to the UK.

It took them a few years to get established there, and then they moved over to France and Germany and so on. So the distribution and the pickup has been obviously then country after country and year after year. And right now, for example, a vendor that's rising, for example, would be Domo in the US. And that's quite a similar example so they just started in the uk um i think last year or this year and um in if you look at continental europe you will not find them at all right now because they're simply not present and not well known and so this is basically the adoption we see here but are there vendors that come out of say germany and europe that are producing regular software that is not perhaps known in the US that we should look out for really?

Yeah absolutely I mean we that's what we see in the BSI is that we still have quite a lot of small vendors that are often only regionally active and what we see is that if we look at the, let's say, business benefit created or the project success, often they are extremely successful. So they really deliver very good projects and good

benefit to the users. But due to size, they are often only well known in the countries they basically are operating in. But for sure, there are some interesting vendors. Interesting. So just before we get on to the BART trends monitor, which is something else I want to talk about, just go back to the OLAP report. So the genesis of what you talk about is the OLAP report. And OLAP as a technology is still there.

You've still got S-Base, for example. You've still got other tools. But is OLAP itself traditional multidimensional OLAP as a technology is still there you've still got um S-Base for example you've still got other tools but is OLAP itself traditional multi-dimensional OLAP do you still see that being used and implemented or or has something else come along to replace that now I mean what's your view on that yeah um I mean what's happening is that we see now a lot of um BI vendors now bring their own database technology. So I think one of the market trends of the last 10 years is that we don't see the market as it used to be

where you had pure front ends, basically tools that could connect to many databases. But what you have today is often a bundle of database technology and then the bi tool and this often replaced the olab databases but um there are there are two or maybe it's only one use case or area where it's still a lot of olab database are being used that's typically in the finance and controlling departments and and then there's one use case that's planning and budgeting where a lot of the solutions in the market are based on

molab technology since it's really very well suited for these types of use cases so who are the i mean i'm aware of tm1 and. I mean, what are the other vendors in that space that are scoring quite well at the moment in your surveys? Well, first of all, these two are still around and both have quite happy users. And then the third big vendor, obviously, is Microsoft. So the SQL Server Analysis Service is extremely strong. It's really, like, can be considered as a standard.

So if a company is using Microsoft SQL Server technology, very, very often they also use the Analysis Services. And I would say these three vendors are really taking most of the market. Okay. So let's look at another thing you've done, which is the trend monitor. I mean, what is the Bark trend monitor? What's the purpose of that? And what were the main, I suppose, the highlight findings really from the last one of those? Yeah, that's a unique kind of research because we simply ask only one question to the participants and that's how do they rate a number of trends we present to them about 20 trends and we let them rate this from zero to ten basically not interesting

at all or to very important for their work or for their company and what we can see here is basically trends from a user perspective. I mean, it's now end of the year, so we'll see in the next week, we'll see lots of prediction and trends for the next year. And typically, we as an analyst would also do that. But obviously, that's an expert opinion. And I mean, often it's right, but it's purely a viewpoint. And we thought, why don't we ask the users what's important for them and what came out is often quite astonishing because um

often users rate trends quite different from vendors and service providers and And since these people and companies and groups also took part in the trend monitor, we can actually compare that. And that's what we can do with that product is we can now run a lot of comparisons. So we break that down by the user type, also business or IT users we break it down by industries we break it down by a company size and we also break it down by geographies and we can also see that specific trends are valued quite differently in different

countries so what comes out here is think, quite a differentiated and very interesting view on the perception of importance of BI trends. That's basically what it is. So I was very surprised to see top of the trend list this time, master data management and data quality management, which is like the equivalent of cleaning your teeth and tidying your room, isn't it? It's something that everyone thinks says they do and is interested in in but i was very surprised to see it was top of the top of the list so tell us about that and and what people were telling you about that topic yeah absolutely i mean it's um i think it's like a rebound it has come back um it used to be a hot topic but quite quite some time ago and i see now with all the initiatives around, for example, predictive analytics and digitalization and the self-service BI,

as I said, the maturity that has been achieved in self-service BI also, we see that a lot of users and a lot of companies are reassessing the value of data and how they treat this asset. And they see that they have big plans and often now digitalization and using data and analytics has become strategic. But once they started into looking how they could do that or how they could improve to use data and analytics, they often discover that the underlying asset, let's say the raw material, is in such a bad state that they basically cannot derive value out of their projects because simply the data quality is not there.

Or they cannot get the consistent master data and so on and so on and i think this is like a sign of maturity that we have now tried a lot of self-service that we see data now as a strategic asset and that we have ideas around using advanced analytics and a lot of people now see the value of clean data, of good reliable data, but they also see that they need to do something about it. Okay, and I also saw high on that list data governance as well, and we talked earlier on about that was a trending topic, I suppose, really, but how is that manifesting itself?

Is it kind of looking at things like data stewards to to own certain definitions of data and what what does data governance mean in terms of a trend and what people are doing about that yeah i mean first of all it's it's more an organizational trend so the the solution to uh that's also the solution to improving data quality and and also to have a better data governance is rooted in organizational measures, meaning that you exactly create these roles and that you finally persuade a business that they have to take care of data. And it's not an IT problem to provide clean data. And this often leads to major disputes and long-time discussions.

But in the end, it's really about organizing it, about creating responsibilities, maybe also sanctions, and to really now get it done. And so organization comes first. And second, obviously, we also see an uptake of tools being used. There are now specific solutions that are getting quite popular around metadata management, around cataloging data lakes, and so on and so on. So it's really also obviously then supporting your data governance initiatives with the right tooling but for sure it starts with more organizational issues and and approaches okay so you mentioned data lakes there i mean do you see much project success or traction with data lakes or is it more of a good idea that perhaps is impractical in purpose. I mean, what success are you seeing with this,

really? Is it happening or not? It's happening, maybe not on the scale that we, especially vendors, thought a few years back, but they surely play a role whenever you have quite heterogeneous data sources and data formats. So it's really a place to integrate that and also to integrate data that is not fitting into a relational database. Secondly, for some companies, it's a way to become more agile because they have much more regulation around data lake compared to a data warehouse so they they manage to get more agile and just quicker to to make new data sources available to users and thirdly where i see it playing a role is really in providing raw data to analysts especially if they run if they are more in a data scientist role so they they really find a lot of raw data from many

sources and and here it creates value so you mentioned about um i suppose raw data there and preparation and so on i mean one of the trends again within the industry has been data prep tools and maybe bi vendors adding data prep um and uh features to their bi tools do you see success with that is it getting traction is there value in that you know what's what's the what's your um survey survey respondents view on that kind of area yeah it's actually one of the trends that has arisen in importance quite a lot so we see that that users really rate it very highly now and think it's an important trend. And what we see, it really has started with the self-service BI tools, because from the very early beginning, what users really valued is not

only to have this flexibility in analyzing data, but also in integrating data for themselves. And often this feature is used for external data, actually, or for data silos that were not accessible before. And so they used it and really liked the features. And very early on in selection projects, we saw that the data integration features played a major role when selecting a self-service BI tool. And now we obviously have these specialized vendors also being available.

And we also see that the data prep capabilities in the cloud platforms are being enhanced. And we see that it's it's um it's becoming a really important feature because it's it allows you to quite quickly access data especially external data um it's often really used to when you're more in an explorative mode when you discover data sources when you just want to take a look whether there's some values or some interesting data and later on when when something is is really meant to be in a consistent production you can still maybe move it to a more traditional data integration pipeline for example but data prep really provides this flexibility and the fast access to data and this has become a more and more important requirement

okay you touched on a topic there was quite dear to my heart from before which was a selection of bi tools um do you still do you see the people's companies still select tools via i suppose rfps with kind of like point you know checklists and so on there i mean how has the selection process itself evolved over time for bi tools and is that has it improved or or what really so maybe um first of all from the from the survey data we can very clearly see that companies that take some effort in comparing the the capabilities and the features of products in in the end run more successful BI projects. So if you look at business benefits achieved with introducing a BI solution,

then we have a very clear correlation that companies that took some effort, that made some comparisons, are achieving higher business benefits. So we can really recommend to do that. On the other hand now, obviously with the the popularity of self-service tools, things have changed. So we see less large, let's say company-wide standardization projects where you would select a big platform for the whole company.

And we have seen more, let's say, smaller scale, more departmental scale selections where you would compare self-service BI tools or maybe compare those with the enterprise platform that's already in place to really understand where are the benefits and what are the downsides of it but so it definitely has changed typically has become quicker it has become more proof of concept based less paper based and this is really because of the the yeah the nature of the tools that are being selected today okay you mentioned earlier on about agile you know it's cloud making you more agile and data lakes making you more agile. But I noticed in your in your trend monitor, the agile BI projects or agile BI development was the biggest fall in all the kind

of the trends there. Again, is that something that was a good idea? Or is it mean it's because it's more commonplace now? What's the what's the kind of the the view on that yeah i think it's it's quite commonplace now so um that would be my interpretation that users are not rated as um amongst the most important trends for them because we see that that most users have implemented it that it's a quite a commonplace way to proceed and and to run bi projects so i think that's the the reason why it's not so important for users anymore because they simply do it yeah it's just become i suppose a given really yeah um so last last question for you so gdpr um it yeah maybe explain for people what that is and and um and why that will maybe be a priority next year and why it will be something

that people looking at analytics projects should be aware of really yeah absolutely first of all it stands for a general data protection regulation so it's a eu regulation and many people don't know that is already in place so it's already oh there we go yes but why there's this magic date of May 2018 around us, because this is the date when companies can actually be fined if they are not complying to the regulations. So that's why it's so important to be up to speed with that in May next year. And the fine can actually be quite hefty. It can be up to 4% of the global revenue of the company. So if you are a larger company, that can be quite a fine if you are not complying with that. So we see that a lot of companies are looking into it right now.

And it's actually for now data and analytics projects is quite far reaching because there are some very interesting demands in here that sometimes are very hard to fulfill. So, I mean, most companies are quite aware about data protection and ensuring privacy, but there are some things in here that are quite hard to fulfill. And one is, for example, first of all, that you have to know where data is actually stored in your company that is related to your customers or to humans, to people. And that is creating already the first quite big problem for many companies

because they often cannot say where the data is. And if we now think back to the self-service BI discussions we had, then it really creates a problem that you have hundreds of self-service BI tools in the company. Each of them has a database and people created their own applications. And probably they also have, for for example customer data in there and you don't know where it is but you have to know that's number one number two is it gets much stricter on who has access so for example you cannot allow people to have a full database access anymore and you you you really have to be sure that you have an identity management in place and that you know how to access data and so on.

So it's really a lot about data management, being aware where data is and who has access to it. And then I think to mention one interesting other thing is that a lot of companies don't see yet but which also has some very interesting implications uh in in gdpr article 21 there is a mention of a right to explanation in automated decision making so this comes back to the discussion we had so your customers have the right to demand an explanation why a certain decision has been taken by your company. And if you now think that you have models making more and more decisions, like, for example, on credit approvals, for example, and then you see that the most popular method or approach right now is using more neural nets, for example, in deep learning. Then we see that this is a black box technology. So basically you train the net, but it's very hard for you to explain later on

why a specific output of a model was created and then we have now the starting discussions whether we can actually continue to use neural nets or neural networks in cases where in the end the customer will demand or might demand an explanation why a certain decision has been taken by a model. And so this right of explanation, I think, has some quite interesting implications in running your advanced analytics and data science initiatives in the future. And we saw that a lot of companies don't really tap into this topic yet

because they are really struggling with the other demands that are quite demanding, obviously, also. So is this a European thing only, or is it a worldwide sort of rule, or what, really? I think, yeah, first of all, obviously this regulation is happening only in Europe. Specific parts of it are obviously also in place in other countries. And for example, I recently heard that in the US also,

if you have credit approvals or credit decisions, that for for example there are no neural networks being used for exactly this reason that it has to be transparent how a decision is taken and you have other methods in your in your library or what you can use for example you can also use decision trees and here you have the same effect you can create a model but this is explainable so we see some things happening also in other countries but right now in the extent i think it's quite unique in in the eu but i guess i guess companies have operations in the eu in the us and have to comply across both of them and you know you're not going to have the same you're not going

to have a different approach to doing cases in the us to europe even to the point of i suppose we had one uh it's issue today with mike where i'm working where um the us cloud vendor for the bi software we're using they were they have to be able to kind of control whether or not us people can log in to do serp to do support work because they can typically log into our environment and do that and so i guess it's something that is you know it, it's kind of known about, but the point you raise there about the understandability and explainability of models, that's a really interesting thing, isn't it?

Because these things, especially we've got self-learning neural net systems where, yeah, exactly, how do you explain it really? It's kind of interesting. No, you can't. So you really have to rethink where you use these black box approaches, like neural networks and where you cannot use them excellent okay well look we run out of time now so um carsten you tell us about how do people access the ola reports how did that sorry the bi survey

and how do they find out about what you're doing and maybe take part in next year's survey yeah that's really would be great if more people would take part um it's basically the central website here is www.bisurvey.com and that's where you find a lot of information we also have a lot of free um documents available and here you can also sign up to participate fantastic well it's been great speaking to you um have a nice evening rest of the evening and yeah thank you very much yeah thanks to you mark Thank you.