Drill to Detail - Drill to Detail Ep.58 'Candy Crush Saga, Analytics and Mobile Gaming' With Special Guest Jonathan Palmer

Episode Date: July 3, 2018

Mark Rittman is joined in this episode by Jonathan Palmer from King Games to talk about the role of analytics in the development of Candy Crush Saga and other King games, their use of Looker along with Google BigQuery and Exasol to provide analytics capabilities to their game designers and product owners and his approach to doing all of this in a fast-moving, technology-driven internet business.- Candy Crush Saga article on Wikipedia- King Games company website- “How King Games is Crushing Games Data” DataIQ article- Looker and King Games case study- Jonathan Palmer on LinkedIn

So hello and welcome to another episode of Drill to Detail, the podcast series about the new world of big nation analytics, and I'm your host Mark Whitman. So today I'm pleased to be joined by someone I've heard speak at several events in the past, being interviewed one of our other guests from the past Daniel Mintz from Looker so I'm very pleased to be joined by none other than Jonathan Palmer from King Games who's a fellow Brit and also probably was interested to see that England have gone further now than Germany in the World Cup and yeah it's a good day today. Indeed it is yes happy days and commiserations to your German listeners. So Jonathan, tell us a bit about the role you do at King Games and how you ended up there. So what was your kind of route into the industry and to working at King Games? Sure. So I guess my career has taken on a number of forms over the years, but I originally started out as a data-focused software engineer many years ago now, too many to mention. And then

that's led more and more into the BI space. And so prior to joining King, I was working as a sort of business systems manager and BI analyst kind of combined role for small intellectual property attorneys. But I was quite keen to test myself with bigger challenges and also keen to get into a more youthful and 21st century technical challenge as well. And that took me into King, which was as a BI developer, first and foremost. And then having been there for five years, I've now found myself as director of core data services there. And it's been a great journey. Yeah, it's interesting. So you came in originally as a BI developer.

What sort of tools were you using then? What kind of things were you doing when you went in there, first of all? Ostensibly, the main responsibilities was was to knock out click view reports um in in those early days i mean i would i joined during the the explosion of our you know king's most well-known game which is the original candy crush game um so really uh building data products um building reporting for a business that was pretty much changing overnight. So going from thousands of players to hundreds of millions of players and dealing with the data challenges that come with that. So it was a really challenging time, a really interesting time.

But, you know, on the face of it, it was still quite a traditional approach to BI, which was, you know, somebody raises a request and says, would like to be able to uh see a dashboard a report about this topic and off you go build some sort of transformation and aggregation and then present that in click view okay okay so what was so when you went and joined there and you said it was a more youthful kind of environment and it was a more 21st century and i've worked in kind of gaming places i think in the past as well i mean tell us i suppose what was different about king and the vault i suppose in the way the way they kind of develop products and the developed games it's very data driven isn't it i mean what's what's the typical process around how a product is built there really at king okay so um yeah it's it you're absolutely right to say it's very data driven you know

the idea with King is that games start small with very small teams of developers and artists and you know the risk level for investment is is pretty low so looking to prototype relatively quickly and get a game out into sort of play testing in order to then start to be able to gather some of the data points you need to see whether the game is ultimately going to be a success or to see if further iterations required to drive it in the direction you want it to go. So that's really the first stage that data enters the equation is when those first play tests occur, when the game is usually relatively limited in its feature set and being released to a very small market. You know, a few thousand users in the Philippines, for example. And at that point, there are no monetization features through so all you're looking at is

how are players starting to play the game how engaged are they how is the overall sort of funnel across the first time user experience what's that looking like and then the the game iterates and expands and improves from there to the point where you're in a position to either make you know the call to invest in it um as a you know a long-term hit or to say okay let's move on to the next experiment okay okay so you mentioned experiments there so do you do a lot of i mean you do a lot of ab testing or a lot of kind of testing in general to sort of see what's the best approach to take and so on i mean is that is testing and stats and so on a big part of what you do there uh massively so so at any given time i would say at king over the last um say three to four years there's been anywhere between 80 and 100 data scientists

employed and they're the chief the majority of those are embedded in the game studios themselves so as you say they're doing a lot of ab, they're doing a lot of A-B testing, they're doing a lot of predictive modelling around what the customer lifetime value of the players look like, what features resonate more, how you present certain options within the game to see which get people motivated to continue playing versus those things which turn players off.

Yeah, definitely. I mean, so, I mean, again, working in that kind of industry in the startup world, I was surprised at how, I suppose, how mathematical and how numerate and how kind of, I suppose, stats aware the teams are there. Is it the case that generally, you know, from product managers or business managers all the way down, you know, people are very, I suppose, KPI aware and numerically aware in these kind of businesses? I businesses, particularly at King. Absolutely. So it surprised me as well.

Having worked in more traditional businesses, I expected people's data literacy particularly to be poor. They're more passive consumers of data, often in the form of an Excel spreadsheet and so on. But at King it's very much the other way around. As you say, those product managers are highly numerate, highly data focused and many of them have come through the pathway of being data scientists or data analysts in the path, in the past. So they're looking to, you know, if they have bandwidth, they're able to dive in on the tech level of the tools. They're able to write their own SQL, for example.

But obviously, once you're actually tasked with managing the product yourself, you want to be doing that less and you want to be using more of the traditional BI suite and so on, and you want to be delegating lots more to the analysts and data scientists to come up with the deep level insights. Yeah yeah so you said when you went there you used ClickView and I mean how did that how well did that work as a BI tool in that kind of environment I mean it's obviously from the world I came from it's quite seen as quite an agile and quite a dynamic and easy to use tool but i could imagine the analysts in a games company would think it's quite constricting what was the how did that work for you really there um yeah you're absolutely right i think uh

in a in a sort of fully centralized bi function as we had then um we felt we were doing a pretty good job and uh it was interesting the the journey to sort of realizing that there were definite sort of categories of users and other data-focused people in the business who were gradually moving away from those tools and gradually becoming less and less engaged. There was certainly a point where the ability, the in-memory exploration element of ClickView worked very well. And there are, I guess, certain key use cases, you know, for example, the centralized macro level KPI dashboard for the business that still fit pretty well within the ClickView stack and where people are still comfortable with using that. But as you say, the more data literate the individual, I think the less inclined they were to use it over time

as their needs became much harder to, you know, the business became more fluid, more experimental, the velocity increased. It became very hard both from a tooling perspective and from an organization perspective to support that velocity from a centralized BI perspective. So that sort of traditional story of here comes a requirement for either an adjustment to an existing ClickView asset or a request for a new report. You wait till that gets to the top of the backlog.

The team goes off. They build the pipeline for that, eventually release it out into the wild, do some testing and iterate on that product. That became extremely slow, prohibitively slow. And as a result, we had to consider other options. And that's for sure. Yeah.

Yeah, I guess that probably was less of a function a fee less of a restriction of say click viewer more about um the way that um probably you know you and i used to do bi before which was you know you would aggregate data you would get averages you'd get kind of you do various kind of aggregations on that data whereas uh the thing that struck me working at qubit for example is, is how the nature of the event level data, which is this person did this thing and they're intended to do that thing. And following that through there, it's quite different to the kind of analysis that we used to do with tools like ClickView. And it was actually Yali Sassoon from Snowplow that made that point a couple of episodes ago. And it struck home, it really resonated with me that that nature of event-based data and the analysis you do is very different to the sort of thing you do with just simple aggregation tools.

Yeah, that's very true. And I think the other side of that is the sheer volumes that you're talking about. You know, in order to make something performant within an in-memory tool, you have to pre-aggregate. And when you're doing that, you're making some prescriptive decisions about what is a reasonable grain to present to the user that does not necessarily reflect the meaningful grain that that user needs to explore too, especially with very sort of wide data. You know, traditionally at King,

you're looking at sort of retention curves and RPI curves, revenue per install curves. You can be looking at the first 30 days are the most interesting part of that curve, but the first 270 days can actually tell you a huge amount as well. understanding every point across 270 days, for example, but doing that by multiple cohort combinations results in this sort of explosion of granularity that actually the average user is interested in a very thin slice across that. But in order to enable the user to choose their own thin slice, you have to create something that's prohibitively large. And I think that's where more the sort of tooling available these days gives you far more flexibility around that. OK, so that's obviously why I heard about you from Looker.

So you implemented Looker at King. So what was I think you've led into that now a little bit. But what was the kind of rationale behind that and what was the problem that Looker was going to solve for you there really that the capability it gave you didn't have before it was a it was a number of things really as a that that's retention curve and RPI curve is it's a perfect example of where we we don't want to pre-aggregate we don't want to um we don't want to pre-aggregate, we don't want to define too tightly the possibilities of exploration that are available to a user. What we want to do is orchestrate the arena of truth.

That sounds like a grand term. It does indeed. So then there's the other factor in our local implementation, certainly, which is to think about data tooling in this sort of decentralized world. So where two studios are ultimately at a macro level, they're reporting on the same KPIs and one game is compared to another at the very tallest level. But in terms of individual game features, a saga-based game, a traditional level-by-level game, a match-three game like Candy Crush is extremely different to a resource management game where the concept of a level, for example, doesn't really exist. So being able to provide data analytics infrastructure

that enables those things to stay as unified as possible up until the point that they truly diverge and then have equal value where they diverge and you can then hand over the keys to the data literate people in those game studios was a really key thing for us with Looker. Because we weren't able to achieve that with other combinations of tools. We either had to lock everything down to keep the macro right, or we had to let go of everything in order to get that decentralized aspect right. Okay. So for anyone listening to the show who doesn't know Looker,

just tell us, maybe just paint a picture really of what the kind of Looker setup you've got is there so you've got obviously a looker model you've got a data source and so on but you know do you have do you have a sort of single model for all kind of games do you as you say you know they vary a little bit how is it kind of architected really so i think um the key architectural decision we made uh with lookers is to take this idea of a single model and then put it into this hub-and-spoke approach. So the idea is that, I guess traditionally you would have a single model for sales funnels, for example, a single model for revenue recognition. What we've done is we've created a core KPI model that is keyed at essentially the device or player grain, if you like. And then we literally lift and shift that model,

defined in the LookML, to satellite Git repos. And that enables people then to use the extends feature within Looker. So they're able to take that code, use it as a base, and then extend it and add their own kind of contextual richness to it. And they're able to do that on an event level because those events are also keyed on the same level.

So ultimately, the sort of core, the hub of that model is the macro. That's the KPIs that apply to every single studio and function. And then once they extend that and enrich it, they're able to get into their own first-time user experience funnels or their own game economy balancing. But all of it joins together extremely nicely and it's built on the same kernel of truth. Okay, okay. So what's the database you use with this?

Is it kind of, I think I've heard you mention Exosol in the past and BigQuery and so on. What do you power it all really with at the database level? So over the last 18 months, it's been chiefly powered by Exasol. There's been a little bit of Impala thrown in for good measure and various smaller MySQL databases for more isolated use cases but we're in the process of migrating the backbone of our data warehouse to BigQuery.

So that will replace XSL and Impala. So what's the reason behind that then? Why would you, I mean, because BigQuery's got its pros and cons. I was kind of curious, why that choice? A variety of things, really. But on the most fundamental level, with our mixed data warehouse approach, the decision anyone working with data has to make at King until now is,

do I use the Hadoop ecosystem for the full rich history of any aggregate or any event level data? And this is a sort of multi petabyte scale or do you use exosol for super fast analytics across a smaller recent subset of that data and that's okay to a point and we were able to make giant strides through separating those two things over the period of time I've been there but with tools like BigQuery now, we're essentially saying, okay, we no need to have to make choices about which SQL engine we're using.

We no longer have to choose between speed and history. We can get an optimum balance of the two. And that's been, I think, the biggest driver. On the side, there's obviously the overhead of running your own on-prem data warehouse infrastructure. It's expensive. It's extremely time consuming. And we have some enormously talented engineers who can do far more than keep the lights on on that infrastructure. So they have a big opportunity and we have a big opportunity to get far more out of them. King, when you take a lot of that overhead away and can focus it on what's more of a competitive differentiator

than just your ability to power on-prem. Okay, do you use things like Cloud Dataflow and the rest of the google stack as well for your pipeline or is it just bigquery um principally it's bigquery but data flow is definitely a an increasing part of the picture but it's relatively early days and so we're still experimenting with what the uh the optimum stack looks like okay what's the response time being like on bigquery i mean i get one again one of the things that surprised me about coming from say the world of oracle i was in was was was

how how easy it was to work with say big query the amount of you know the fact it's almost like no ops it's just there and it works but you know you still do hit issues where you know joining large tables are going to be slow have you found the response time to be on BigQuery compared to, say, Exasol, really? I think there's a few ways of looking at it. I mean, Exasol is extremely quick. For those who don't know it, it's, you know, columnar in memory. When you have a well-constructed data model and you point looker at it, you can expect, you know, sub-five-second results, often sub-one-second results with some of your analytic workloads. You do pay a bit of a penalty with BigQuery.

There's a few seconds overhead on any query. much larger datasets, we certainly see performance that is at least comparable with Impala, which many of our downstream data warehouses like data scientists are more than happy with that when they're querying several terabytes of data. That said, as you mentioned, thinking about joins is a key thing. And so we're, you know, as we've taken the approach that the migration will literally lift and shift as is our data warehouse. But we've got a parallel track of work of starting to think what, you know, what that really should look like in a BigQuery world once that migration phase is over, experimenting with, you nested uh nested rows and so on that are quite a differentiator of the products and from the early experience we have those are quite

mind-blowing in their in terms of their possibilities okay okay okay so so i guess the other thing i mean the other thing about looker that's interesting is that is the lookup is the semantic model the looker model and certainly coming from uh i suppose some of the more some of the more kind of desktop tools that have been popular recently like tableau and so on that don't have those you know you can see the value of them really um but i mean how have you found that the value of that has been at king i mean putting the idea the idea of having a business model of what you do and that being something that can be extended and and and kind of all that how have you found that really yeah i think that's the the really the secret source of the looker offering that that it's it's enormously powerful i mean we uh in our legacy stack we

pushed as much business logic into the data warehouse as possible it was mainly described through views um and that worked to a certain point. But you know, as Daniel Wintz from Looko would say, it's very hard to reuse that code and it's very easy to lose track of that from a data provenance point of view. Looko, that semantic layer has various benefits, but one of them is just is just a transparency factor um you know the fact that a user who's sql literate for example can click on that sql tab and see what's actually been generated uh to to what's actually been used to drive the results of the the visualization they're looking at um and that really wraps up a huge amount of the overhead on more traditional

BI teams. I think, how do I get back to that number is a question that we get asked a huge amount, well, less and less so these days we look at. So on the one hand, you've got the transparency and on the other, as you say, you've got that extending factor. So we really control that semantic layer. We can really describe in a clear, reusable, and important source-controlled way. But that doesn't come at the cost of agility. And so once people can start extending that, they can add their own business logic. And that, in turn, is transparent. Okay.

So you mentioned about metrics and so on earlier on. I mean, have you found the process of trying to get the idea of there being a single version of truth? it's transparent okay so so you mentioned about you know metrics and so on earlier on i mean have you found the process of trying to get the idea of there being a single version of truth i mean if you pick one of the key measures like engagement for example you know has it been you that's had to kind of get that defined as a single definition or has the business done that or is it not is it just not natural you can get that to be um to be kind of commonly understood across the whole business how does that work in a gaming company like yours um i think that there's been a huge appetite at king to to get that right and uh to speak a common language so i i certainly know from my own

experience in the past where you you sort of have to uh bang your head against a bit of a brick wall to get that work done but But at King, there's always really been that culture of making sure that we're converging on a similar understanding. I think by contrast, what you tend to get into is then more nuanced conversations around some of the more fundamental things about what you're measuring. So whilst everyone will understand that engagement, for example, is usually looked at from the point of view measuring players can be extremely difficult because they could be playing across multiple devices they may or may not Facebook connect for example so de-duping at that level to understand

a true sort of player lifetime value versus understanding, for example, when you're buying installs through Google AdWords, for example, you are looking at that more from a sort of device-centric point of view because you're ultimately paying for the install on that device. So, yeah, that's where we tend to get into the more nuanced discussions but at the top level um it's been one of the easier parts of the journey to to look at what the sort of core kpis are which means we've been able to focus on how we use the tooling to articulate that okay so the users of looker in king games i mean you've obviously i presume you've got people who are just consuming dashboards and so on there i mean but do you have people in who are in other

users who actually would write looker mail themselves and do more complex analysis like that how's that worked out for you um yeah there's there's sort of a spectrum um as you say on the one hand there's a more passive consumer maybe someone who doesn't even actually ever log into looker they're receiving something that's scheduled in Looker in their inbox every morning and that's fine. Then there's probably a very substantial cohort of people who are using dashboards that have been built by other people for them and they're just filtering and occasionally drilling into that little. And then there's a fairly sizable cohort of people, I would say, who go, you know, think about things from the explore onwards rather than from the dashboard back, if that makes sense. So

they have their explores they go to every day and they're happy to build up their own dashboards, their own schedules, and really understand the power of that ecosystem. And then there's a smaller cohort of people who are writing LookML. So we still have a semi-centralized data function at King. So we have both core teams that define that core macro level KPI model. Then we have semi-centralized teams who specialize in vertical verticals so a team that deals with marketing and ads for example versus a team that deals with CRM and then further beyond that we have the the game studios themselves who have their own data scientists and increasingly

their own data engineers as well and so they're both extending those core models and they're building their own look ml models from scratch so we really have a we have many of everything i would say from a from a king perspective so you really get to see all sides of the product um and its strengths and weaknesses okay okay so so again when one of the things that i found different about going to a tech startup, it was the role of analysts there. We had analysts in the company I'm working at now where they wrote things in SQL and didn't use BI tools. And that surprised me in that, you know, I used to write SQL years ago, but generally we'd use tools like ClickView now. But actually, you know, the ability to kind of handcraft queries yourself and construct them in a way that follows this kind of, I suppose, the event model and what people are doing and so on.

I can see why they did that. And one of the things I was expecting was that they would perhaps graduate from writing SQL to writing LookML instead. So an analyst would write LookML and would write queries against the model that way at a level of abstraction higher than sequel but never really kind of happened really did you find that happened at king or did did did people who wrote sequel carried on writing sequel and and look at mail was more your team really um no we i definitely recognize that um there there are some people who we've not been able to take on quite the same journey but increasingly they're in a minority. And I think that the trade-off and the message from us around Looker

and why we thought it was so important that people made the most of the platform was really around the fact that, particularly in the game studios, and it was ultimately we were thinking about the studios most rather than thinking about centralized functions when we when we invested in looker um you know one of the recurrent criticisms and i i'm pretty sure king's not unique in this respect is employing data scientists with the you know phds in machine learning um and them being the de facto data literate person in that team who's then tasked with coming up with fairly basic KPI reporting in some form or another. So the trade-off to them was to say, look, you know, you're probably never going to be able to shake that element of your role off entirely. But what

you can do through Looker with this abstraction is to reduce the amount of time it takes you to replicate and iterate those more mundane KPI tasks tasks and so the people who sort of grasp the nettle there have really then gone on to realize the broader benefit of of abstracting their sequel work but that had to be the the sort of sales pitch if you like had to be more around what they found frustrating in their day-to-day role that the looker was going to help um but yeah there are definitely some now who um who who start from a looker first mentality which is really nice the utopia for me is as you say that point where everybody thinks look look at first and then really only sort of dives in uh when when it's something that there's there is no point in abstracting you you could truly hand on

heart say that it was something that could not it would not be done again or was a truly disposable task. We've not quite got to that point, but the momentum is in a positive direction. Okay. So you mentioned source control earlier on. How important was it for you that, for example, Looker connects to GitHub and it follows that, I suppose, software development best practices that people do these days. Was that an important thing for you? Massively so. We tried various attempts over the years to get our ClickView stack to conform to those expectations, especially when you use company IPOs, for example, that sort of thing becomes, auditability becomes extremely

important. And we came up with some nice solutions to that, which everybody was very happy with. But it was, you know, it was not a natural thing to do, if you see what I mean, in the context of that ecosystem. So the fact that there's just an innate part of the product in Looko is a big differentiator and I think also when you're when you're dealing with that broad spectrum of you know there's the passive business user they don't really care about that but when you're dealing with a business with so many data scientists and so many technically savvy people they really are quite happy in my experience to make sort of face value judgments about how 21st century

a tool is, for example. That's one of the features that we, to any technical audience, make a big point of emphasizing first and foremost. This fits in with your natural data engineering workflows. We followed we've we've we've followed some classic software engineering practice through to a point and then the bi tools take over it's it's really you know a seamless ecosystem and again the sort of transparency of uh that comes with that is is crucial yeah i think that that kind of cultural fit is important isn't it the fact that the fact that they feel comfortable using it it's using apis it's using sort of github and all that kind of stuff it means the tool gets more just ready acceptance but then i mean you mentioned there are data engineers and and and so you must have

obviously quite a kind of a team of people behind you that are building this out and supporting it what's the setup really in terms of data engineers and the supporting staff that support what you're doing with kind of Looker at the moment? Actually, the Looker element is one of the lightest pieces from a centralized function that, you know, I'm director of. And that, again, has been one of the big advantages, really, is we've been able, where traditionally we have these sort of big centralized teams, we've been able to devolve so much out of it to the business that those engineers can focus on other things. So our actual team that led the Looker rollout at King and took it from zero users to, I think, 700 odd at the latest count,

that was two people, myself in my previous role at King as BI lead and one of my teammates. And we were able to drive that journey of user engagement and onboarding and delegate out the model development to the studios themselves, but providing that sort of those core foundations you know that the the the interesting thing for me around this is that the core model that i wrote in the uh with my colleague in the sort of summer 2016 has hardly changed at all um in that two years now um and yet there are hundreds almost of satellite models that consume those foundations and go off in all sorts of different directions and still do so to this day. So it's quite a quantum leap in terms of how we thought about how to structure things. And there are lots of teams building semi-centralized models as well. But I guess, again, they find the same thing

that actually the upfront work is fairly light and then it's where you take it to the next stage that is then cooperative, if you like, with the downstream teams. How did you find understanding how Looker works at the start? I mean, when I first encountered it if you like with the downstream teams how did you find how did you find understanding how looker works at the start i mean i i when i when i first encountered it and we had this concept of explores and views and and and all that kind of stuff yeah it didn't seem that having come from

a world of of subject areas and dimensions and measures and so on it kind of seemed a bit odd and the wording that they use as well with kind of explores and looks and so on is a bit kind of different how did you how did you find learning that and how did you make that kind of mental jump to how the correct way of doing look and modeling and the correct way to structure things really um yeah it's definitely you know it has its own vernacular doesn't it and um it takes a little while to get your head around that you know do find the view term to be confusing. Is it a database view? Is it a table? It's strange that they sort of use the reserved word, if you like, for one of very intensive sort of offsite

with a couple of very talented Looker engineers, three of us at King, two talented Looker engineers, and we locked ourselves in a meeting room for five days and didn't leave until we built a couple of core use cases that we could then use to lead our pilot project. So by the end of that five-day period, we thought about very little other than Looker over that time. And so it very much got up to speed with it. But I think that said, it turned out to be a fairly quick process because we realized increasingly as we went through the journey that the way that the product has been thought about very much mirrored how we've been

trying to do things with other tooling anyway and try to bend it to to to become more lookalike we just didn't know that there was something out there that was already doing these things until until that point okay so so do you tend to sort of join everything together it will your views together in one explore or do you separate them out how do you if you're guiding your team members into how to design a good and efficient um look or explore yeah what are your kind of i suppose guidelines that you found have worked really in the in the past um there's a there's a few key things um there's uh there are certain topics where um upfront materializing and aggregating is not prohibitive so that certain you know patterns

that never really change and so we we always try to point out the flexibility to say that this just because you can join everything at the very finest grain and it can roll up and aggregate and and smartly understand that joining it you know at query time doesn't mean you should so we try to encourage people to think about what can be materialized up front to help performance um and we try to think, get people to understand the possibilities of caching as well, the data group element to think about, you know, what really is a brand new query in a given day versus something that, you know, you know that someone's only going to answer that question once in a day or a hundred people answer that same question. So, you know, do it once and

heat that cache and then go from there. So, there's definitely, once you get into the weeds of things that people have become really comfortable with the tool, you can really take them through a quite sophisticated architectural journey, I would say. But up front, you know, what we're trying to, you know, expose through the tool is is that um you know our events keyed at this on this unique key um and you can through this model uh if you're you can join in as many of those events as you like um and then you just join them to the core explore so there there is a single explore that people are joining to first and foremost then once they're comfortable with that then they start to sort of realize the benefit of okay actually now i can start to create smaller more niche explores that maybe deal with a single av test for example that doesn't need to

traverse that core layer interesting interesting i mean so so i mean you said earlier your model hasn't changed much and certainly looker itself, I think, has stayed fairly constant. The core idea is still the same. The product is still the same and so on. But is there anything that you'd like to see added to it, or you've seen maybe mentioned in a roadmap talk or something, that a new feature maybe in Looker that would be particularly interesting or useful to a company like King, for example?

There are definitely a couple of big ones that I think will be quite changing. I mean, one is a feature that's already in beta that when it's, you know, really available production standard, I think will have a big uptake at businesses like King and that's the ability to, the merge feature. Yeah, I thought you were going to say that. Yeah, that really is a big leap to plugging some of the gaps that still remain from the ClickView way of thinking about things.

So that's super powerful. And then I think actually the things that concern me more and I think would really take the product on to the next level is content discoverability. So, you know, the advantage of our Looker model, the hub and spoke model is, you know, literally any data-driven team in King can do anything with our Looker platform. They can consume from central models. They can explore dedicated models around subject areas. They can mix and match or they can produce their own. And what that could lead to is an explosion of content and an explosion of explores. There are actually relatively few users, King and I guess an awful lot of other businesses who are genuinely interested in everything that's on there.

The front end element of the tool I I think, can definitely move forward in terms of guiding people towards the things that are really relevant to them. Yeah, I think at the conference we were at last year, they talked about having something that would maybe kind of highlight which fields were used with the field you've selected, for example. So if you have got a big model, you know, you might pick one and it would say these fields are also commonly used with that. I mean, that again might help to guide people, I suppose. Yeah, yeah, absolutely. Exactly. That sort of intelligent guidance can happen on really any level of the product. Is it, you know, which are the dashboards that my peers use so I don't need to reinvent the wheel? Versus, as you say, what are the common combinations of questions, dimensions

and measures that are asked of this particular explore. You can really take it down to the finest point and then I suppose you could theoretically take it a step further and take that to the LookML layer and when someone's potentially building something you know have a level of intelligence that we're able to say, you know, there's a possibility something like this already exists. But that's even further down the line, I'm sure. Okay. Do you ever use things like data actions and the Action Hub now as well? Is there any links back to your main applications or is it all pretty much standalone really? It's a small amount of utilization of that sort of thing.

But there's a big appetite for making more of it. And I certainly think one of the biggest users of our Looker platform is the performance marketing team who do an awful lot of work with tools like AdWords. So they're very hungry to get that sort of stuff up and running. And we're certainly utilizing the ability to schedule things to webhooks and so on an awful lot, which is more or less a proxy for those same data actions, isn't it? Okay, okay. So in your mind then, what's the next thing? I guess you've thought about this quite a bit, but what's the next unsolved problem in this area really? What is analytics not yet delivering on its promise for or whether it's an opportunity that you can think of um well it actually sort of taps into something you were

saying earlier about those that those users who who don't necessarily think look ml first and about that abstraction taking that a step further i think one of the things that really strikes me with a lot of contemporary data-driven businesses is increasingly data science and machine learning and so on is a part of the product itself. And then increasingly business expects to have access to that sort of talent, data science talent and that sort of leverage. And then you've still got the more traditional BI use cases. And I think the point at which they all intersect is the bit that I think that that's what's yet to come. To give you a sort of practical example of what I mean by that, if you have a sort of a predictive model, there's obviously an amount of data that goes into that

predictive model, and there's some data that comes out. And actually, that same Looker conference we were at in San Francisco together, there was a great talk about this very point is positioning Looker, for example, within that workflow of, well, why not enable the business user to explore the data that went in and then explore the data that came out and then make that part of a rich ecosystem that enables you to loop and iterate and compare and contrast. And so I think that the tooling and the mentality is there. I've just not seen firsthand yet anyone sort of really put their money where their mouth is and make that leap to saying, okay, let's make this all one and the same ecosystem. Everything's explorable. Everything is able to bring, say, the machine learning workflows within the same uh within within

within a single step okay it's interesting so what for you then what's the future for you really you know what what would you like to be doing in the future and you know where we might where might we see you speak or whatever in the future and so on uh so my my my time at king is actually sadly coming to an end it's been a an amazing journey and I'll soon be moving on to taking on a new role as head of BI at a fintech company called GoCardless. So I'm sad to be leaving King but super excited about what lies ahead. And I guess for me, there's always the thing in any technical role, maybe any role for that matter. But, you know, if I were to go back in time five years and do it all again, knowing what I know now, what would I do differently?

And I guess this next role is an opportunity to find out. Yeah, fantastic. Well, it sounds like you've done a pretty good job at King, actually. So, I mean, you know, you've obviously left a good sort of legacy there. And I was really keen to understand how you've made such a success of it. And I suppose how you've used Looker as well, because it's, you know, it's got a lot of promise. I think it's a new thing to learn. I think it's the idea of bringing, I suppose, a tool into it, a BI tool into startups on tech companies where we can

start to, you know, bring in some of these ideas about semantic model, single version of the truth, but in a way that aligns with the corporate culture and the way people do now, do things now. And he's really good.

And, you know, you've done really well there. So it's a good luck with the new job. Thank you. I hope that works out well. And it's been great to speak to you and yeah, have a good evening.

And thanks so much for coming on the show. Thanks very much, Mark. Take care. Thank you.