Drill to Detail - Drill to Detail Ep.50 'Agile BI, Karl Marx and Our Man from Moscow' With Special Guests Stewart Bryson and Alex Gorbachev

Episode Date: February 27, 2018

Mark Rittman is joined in this 50th Episode Special by our original guest on the first episode of Drill to Detail, Stewart Bryson, to talk about developing agile BI applications using FiveTran, SnowflakeDB and Looker and his recent work developing a BI solution for Google Play Marketing using Google Data Studio and Google Cloud Platform. We're also joined later in the show by Alex Gorbachev from Pythian, our mystery guest who Stewart then interviews flawlessly armed only with a set of questions given to him as the guest was unveiled ... though be sure to listen past the final closing music for the bonus out-takes.#115 Google Play Marketing with Dom Elliott and Stewart BrysonThe Next-Generation Jump ProgramThe Data Sharehouse is HereFrom Data Warehouse to Data SharehouseAlex Gorbachev profile on Pythian.com

So hello and welcome to the 50th episode of Drill to Detail, the podcast series about the world of big data, analytics and data warehousing in the cloud. So my name is Mark Whitman, and I'm very pleased to be joined in this special episode by the first ever guest on the show, Stuart Bryson, who came onto the show again last year and joins us once more for the 50th episode. So welcome to the show, Stuart, and how are things over in Atlanta? Things are going great. We've been busy, growing as usual, and making our way into different technology stacks, and that's always interesting. So it's been a lot of fun. Good, excellent. So, Stuart, so as this is our 50th episode, I'll be going to do something a little bit special later on.

For the first time ever, we're going to bring a second mystery guest onto the show, who you don't know who it is either, but actually you're very kindly going to interview them for me. So you're set for that? All set. As I tweeted, I think, earlier in the week, I feel like General Custer asking for directions on his way to Little Bighorn.

I hope that this goes well. We'll see. I got the questions. I'll make sure that the guest knows that these are your questions and not mine. So Stuart, so there probably are one or two people that don't know who you are still. So do you want to just explain, you know, what, who you are, what you do, and I guess how we know each other? Absolutely. So I've been a colleague, I was a colleague of yours for, I think, six, six years at the old company we worked for. About four years ago, I started a new company called Red Pill Analytics. We were pretty much just an Oracle BI and data integration consulting company at that time.

About two years ago, we started really pivoting and going after the public cloud, really, and focusing there. And that's what we've been doing for the last two years. About 40 or 50 percent of our revenue comes from public cloud implementations, and we're pretty proud of that. Excellent. So Stuart, I always remember you as the person that would argue with me over absolutely every single thing that I suggested. Couldn't I say the same thing about you, Mark?

I know, I know. It takes two to tango. We're good friends, but I don't think we ever, ever actually agreed on anything ever. But I suppose that's the kind of essence of a good working sort of relationship, really. That was good fun. Yeah, I think the solution was usually somewhere between the two of us and probably better for it. Yeah, exactly. So I'm still not entirely signed on for Agile, to be honest.

We'll get on to that later on maybe in the next bit. So that's still something I still couldn't bring myself to do. But anyway, so Stuart, so a couple of reasons I wanted to get you on um on the show apart from the fact obviously you are our first guest and uh you're always uh always good for an argument in a podcast which is good um was you a couple of things going on that theme you talked about there um a couple of things you've been doing recently there was an event that you ran recently with um some of the vendors that I've been talking about a lot on this show recently. So, Looker, SnowflakeDB, I think were there as well and so on. And so I want to

talk about that really and what led you to that and how you kind of, I suppose, how you transitioned from doing that event a few years ago based around the kind of Oracle technology we used to use to more of this new world serverless, as a service kind of technology. And then also I noticed you hid it well, but you did some work at Google a while ago, you and some of your colleagues, which is fantastic. I mean, you tweeted about it the other day and I said, officially, I am massively jealous because you were going into Google, delivering BI for them, using their products.

And how fantastic is that so I want you to talk about that and what was involved in that and I suppose two things from there what was it like delivering BI in Google well that's interesting but also how did you approach it especially with again these new world technologies everything by API and all that kind of stuff certainly I think they're two I, practical examples of kind of a lot of things we've been talking about on this show. So let's start off with, first of all, with this event you ran then. So tell us about the event you ran and just very high level. And what were the vendors involved there?

Absolutely. So we call it the Jump Program. It's a play on the Matrix reference that our company sort of has in its branding. And that was, you know, when we ran it several years back, we were focusing more on agile methodologies using Oracle BI tools, which don't necessarily lend themselves well to that. So the sort of the pivot, we kept the name. We had some marketing behind that, keeping that name. But instead of really focusing on agile delivery with a big A, so to speak, we were focusing more on just being agile in its true meaning, which means being reactive to environments. And so we thought the name played well again. So it was really more about, you know, what can you do in a fast amount of time with very little provisioning? So a lot of these tools, so it was Snowflake as the data warehouse that we used.

We used 5Tran to replicate an on-prem Oracle database to Snowflake. And then we used Looker, the Looker analytics and BI tool, the cloud-based analytics and BI tool, to report against that Snowflake replicated data set. All of this was done live. So we started with no data in Snowflake replicated data set. All of this was done live. So we started with no data in Snowflake at the beginning of the morning. We replicated it over. We demonstrated the functionality of Fivetran. We demonstrated functionality that Snowflake has, specifically their data sharehouse capabilities, their elastic sizing and resizing. So we ran, executed some workloads that caused the, the cluster size to expand and contract.

And then, uh, uh, and then also, um, uh, demoed their, you know, sort of zero copy cloning capabilities they have. And then we moved on into Looker and we had, uh, you know, actually had the, the attendees develop some LookML. We started with a LookML Explorer that was already mostly done but needed to be extended. So the attendees got to extend that. And then from that extended LookML model, they went in and built a custom dashboard from scratch. Okay.

Okay. So, I mean, we'll get on to the detail of the kind of products in a second. But so you're still going down the agile route then, which is interesting. So I kind of, a comment I made about agile projects a while ago, it strikes me as like communism in that, you know, real agile project. You basically called me a communist on air, didn't you? Well, whenever an agile project goes wrong it's typically because the kind of the the answer is well it wasn't agile enough you know it wasn't

true agile and and that reminds me of like countries that had communism where it goes wrong and then basically it wasn't they say well it wasn't communist enough and then everyone dies at the end and it's just tragic you know and i i kind of what i would say to to counter that is i mean are you are you i suppose in a way are you you the kind of Karl Marx of agile in that you do do it properly? Or is it is it is it a bit of a scam, really? I mean, what do you think on what's your considered view on lowercase, lower letters agile now then? OK, so we certainly do believe in it when when when we have a service calledacity Analytics, and that's our development on demand service. And about 60% of our customers engage with us that way. And we actually sell it. It's a monthly service. Hopefully, we sell it in larger increments than that, but you can buy it just a month. And we actually sell it in terms of story points. So it's not about how many hours you get, it's how many points you get that month. So when we're doing that service, we're very much about agile methodologies and boards and all that sort of thing. But we do recognize that we have a lot of customers that

simply can't or won't, or it's not applicable for them. So we definitely aren't communistic about it or in that way. But I certainly do believe in it, even for, you know, for BI. And especially as, you know, when we start looking at newer technologies, there's a lot more code that's involved in delivering analytics projects today. Not necessarily, but if you're looking at newer tools, some of the things you talk about, data engineering, data science, et cetera, there's a lot more code that's involved today. And Agile really did grow up around real code development. So I think it's more applicable than ever there. But if you're using a set of tools or if you're a sort of a legacy team, and I don't mean that in a bad way, but if you've been using legacy non-source, non-code tools to build analytics and data pipelines, then it might

not be right for you. We do believe that there are lots of little lessons that you can learn from Agile that you can apply to almost any project management philosophy. But in this jump program that we ran, we didn't do boards. We didn't do any of that sort of stuff. So we were very much approaching it from a little A agility perspective that just being quick, iterative. I think there's a lot, being iterative, refactoring, not trying to solve all the BI and analytics problems in one go, but solve them a little bit at a time. And if you look at replicating data from a source system and going after it directly, you know, without transformations, without ETL, you really do have to be pretty, pretty agile, little a agile in your approach to that so do you think do you think now that bi has become more decentralized it's being more done by users themselves do you think that an irony of that is that is that actually the whole talk of methodologies has gone away and that

people now just get on with it and do stuff and it's rare to get a bit more than a couple of people really for more than a couple of weeks doing a piece of work it did today to those bigger projects formally organized in that way still exist? Well, they do, and we see them. But when you start talking about proper scrum, proper, you know, pick whatever Agile methodology you want to use. We don't see that happening a lot. But we do see the principles of Agile using boards. You know, all work is logged with an issue on a board. Boards are moved between stages. Code is committed, hopefully, if you're using source control. And so, I think that's more valuable now for a decentralized program because, you know, especially with the sort of the whole concept of microservices now is that, you know, you don't check in your entire enterprise code base into one repository anymore, as we used to in the old days.

Now it's a whole lot of little Git repos running around everywhere. Perhaps there's a Git repo per team. And I think that just the basics of running a board, doing burndown, knowing where you're at in the middle of the iteration and so on i mean they're they're kind of all quite atomic technologies that don't i suppose in a way they're obviously not from the same vendor but they integrate and so on there um we'll get onto that in a sec but you know we used to talk about that as being something we could do with more monolithic technologies like say oracle bi and so on there um i mean obviously everything's an evolution but would you still advocate using agile methods and things like the extreme BI approach that we talked about in the past,

which don't ever look up on the internet because it's not the same as what we talked about. But would you advocate that still with things like, say, RAC, for example, Oracle Analytics Cloud, or is that really only really practical with some of this new technology? So, you know, I'm not trying to plug our technologies, but we did write a product called Checkmate, which was trying to solve some of these issues.

And by the way, that product is free now for anyone to download. So I certainly do think it still applies and that sort of concept, if you're using Oracle technology, certainly applies. But in these newer technologies, you're not buying into a particular cloud or a particular cloud vendor. You're buying a service. And I think that the idea that Fivetran knows it's not the entire solution for analytics. Snowflake knows that it's not the entire solution, and so does Looker. And I think the idea that these companies all support, Looker supports BigQuery, Snowflake, Redshift, and that doesn't keep Snowflake from partnering with them. So I think the idea that your technology needs to support a lot of things is a good thing. You're able to plug in different technologies that suit your needs instead of trying to decide which one vendor am I going to go to

to buy an entire stack. And the fact that all of them sort of approach this from REST APIs, integration, sort of standard integration techniques is really where perhaps you're better off now that these vendors are trying to only be a small part of the puzzle. Well, the problem, I think the problem with, let's name names, the problem with the Oracle stuff we used to work with or any other vendor that would have massively integrate their BI and their data integration tools with the rest of their stack is actually the things

they integrated in weren't the things that we were interested in. They were more concerned about being integrated in with kind of, I don't know, SOA suite and stuff like that. Fusion middleware, right. And actually there was a kind of an overhead, a tax to pay with that really, because it would slow down releases, it would do a lot of stuff that was not relevant to us.

So actually a lot of that integration wasn't relevant to the sort of projects you and I were doing, whereas the integration you get with, say, I don't know, Stitch or Fivetran with GCP or stuff like that, they're actually relevant. They're actually relevant integrations. Yeah, it's about dealing with data. It's not about dealing with middleware, right? You're exactly right that trying to put all of these products in a single unified sort of Java platform wasn't necessarily valuable to people who were trying to analyze data. And so I think that now the sort of the plug-in capabilities between these products are really about sharing data, not sharing infrastructure. And that's where the real value comes in, I think, is that we're up and running, and that's what we hope to get across to our attendees of the Jump program.

You're up and running quickly because all of these things know how to talk to each other, but they know how to talk to a lot of things, and that's really their strengths. Yeah, exactly. So tell us, for anybody who doesn't know what Fivetran is, just explain it in the context of what does it do? But how does it relate to the tools that you and I used to use? And I suppose, what role does it play within this kind of stack? Certainly. So I do want to do a quick shout out to all the vendors. Fivetran, Snowflake, and Looker were all sponsors for this event.

So we were very thankful for that. And then they also sent representatives to take part in it. So we were really thankful for that. So Fivetran is a replication technology. It's low provisioning replication technology. So if you think of other, I won't name names, but if you think of other replication technologies that you and I have both used over the years, it's a heavy IT involvement usually.

There's things to install on servers. There's connectivity between servers that has to be configured. But when you look at Fivetran, it's really just a web GUI interface where you connect your Snowflake instance to an on-prem instance. And what we actually used was an Oracle database. And it was running in Amazon RDS, but in all, basically, in every way we went after it, it basically mirrored an on-prem Oracle database. And so Fivetran replicates that. It doesn't try to be a transformation engine,

really, it does some small bits of transformation, but it tries to deliver the schema, pretty much how it how it sits, especially for Oracle, how it sits in the in the on prem source system. Is that enough, though? I mean, I remember I interviewed Fivetran a while ago, and I was trying to puzzling out during the interview, was that what was the the unique value they brought and in the end I kind of concluded it was the connectors which other vendors like say stitch um think of a commodity if I've trained the ones that actually think there's a real value in that and they put a lot of effort into those into those connectors but as you say um they don't really do anything around transformation so what is that really where's the value in that for most people when when most connectors these days

are free and generic well so so i think i'll i'll come back to this sort of the agile spin on this a little bit which is you know if you replicate your source system schema to a data warehouse you can knock out 60 of your requirements that way. So why must we sit and wait for endless amounts of ETL transformation logic to get the other 40% when we can just replicate the schema, do our transactional reporting, and a lot of our sort of strategic reporting as well that way? And I think, you know, Taylor was the, was the attendee at the Jump program. And he had this question from the audience. And he said, you know, we're not trying to solve all those problems. We're trying to solve the ones that we can solve effectively. And yes,

you probably do for some of some of your real content curation, you might have to investigate doing some sort of data transformation. But that's where Looker came in as well. So Looker has a metadata model. So if you've been using a more of a legacy or big box BI tool in the past, you appreciate what a metadata layer can give you. And so Looker is invested in having a metadata layer that's much easier to develop than I would argue that some of these tools from the past, it's much more flexible. So that's the other thing that we were trying to demonstrate is that if Fivetran can give us 60% and the Looker model can give us 20, 30% more, we're really getting close to full coverage there.

And so why do we, why, and let's ETL when it's necessary. That's one of the sort of concepts that I tried to preach back in the Xtreme BI days, was that we're not necessarily saying that you don't need ETL at all, but it's not necessarily the thing that you roll up your sleeves and do on day one so so i've got a metaphorical box of t-shirts in my office with stream sets on the front and and your photo on

the back of it uh because you know you've been quite a big fan of stream sets and and they um and uh and they were quite kind of when i came on the show before they talked a lot about um data drift and how we deal with evolving schemas. I mean, given all these bits you stitch together yourself, how do you handle change over time and the concept of data drift and just the inevitable kind of velocity of change within these environments with these kind of tools that aren't integrated by themselves? Yeah, I mean, you've got me there. There's not a solution for data drift here.

Now let's sort of... Define what that is, first of all. Yeah, exactly. So data drift is the idea that you have columns that change their data type, their length, their columns are added, columns are subtracted over time in a source system. And if you talk about from a stream sets perspective, their tooling really supports the idea that you'll get data drift. And if a column has been added, it's not going to break anything. If a column's been subtracted, it probably won't break anything if you've written your pipelines correctly.

So their idea of flowing data between, you know, sort of mostly big data technologies, but cloud technologies as well, is that you can do a good job of supporting data drift. Fivetran is not there. If the model changes, they're going to go make changes to their APIs and their connectors, and they're going to deliver those new columns, those changed columns, et cetera. There's an auditing schema that Fivetran has to allow you to track some of this. But in general, you're going to have to roll up your sleeves. Now you can do some things in Looker that isolate you from these data drift changes as well by only referencing columns that are necessary for your model, etc. So that if a column's been added, it's not going to break anything.

If a column's been subtracted and you're not actively selecting that column, then nothing's going to break. But there is more work involved if you're not using a data integration technology that truly supports data drift. Okay, okay. So let's get on to Snowflake then.

So Snowflake, I've gone from bemused. I've noticed. So Snowflake, the thing that struck me with Snowflake was two things. One was the practical difficulty at the time that I was having with moving a kind of traditional data warehouse workload into BigQuery. So a lot of the activities and the transformations that I was used to doing, even things like, say, state change dimensions, was a bit harder within BigQuery. And I got that.

And also, when I went to the partner Looker event, and Kevin was there, your business partner, it just struck me how much of the Looker implementations were running on Snowflake. And there's obviously something there that is is resonating with customers and the only thing I remember so Snowflake introduced this thing called a share house so that yeah it keeps my kids remind me that thing that Donald Trump said about about the African countries in that but it's a share house it's a share house not a whatever house our gritman's comments are his own exactly so but tell me what is a data share? How can we get this right? What is a data share house? And why is, what is a data share house? And why are, why is Snowflake so popular in this kind of world, really? So a data share house is, is the concept that we, we don't need to necessarily be moving

large amounts of data between platforms. If you think about like sort of the old days where I had my Oracle database, you had your Oracle database. And if I had some data in there that I needed to make available to you, perhaps we could create a DB link between them, for instance. But the security aspects, DBAs have never been very happy with that solution. So usually what would happen is one of us would write a data integration process to move data from my database to your database so that you could have that data available. What Snowflake realized was that they're running a cloud platform where all of their customers' data basically sits in the same architecture. And they could make data available between customer accounts. When I have a Snowflake database, I don't have my own server.

I'm not running from a storage perspective, right? They've separated compute from storage. So my storage is not sitting in a local storage on a VM. It's sitting in a local storage on a VM. It's sitting in a shared storage architecture. So it becomes very, very easy with just really some metadata pointers and security privileges being granted, et cetera, for any Snowflake customer to query any other Snowflake customer's data without having to move it. And so that's, I mean, I just think about this from the perspective of a company where they're selling data as a service. They could easily build that data in Snowflake, and there's customers that are doing this, make that data,

build that data once in Snowflake and just share it out to other Snowflake customers. And there's really no architectural difference between querying my storage and your storage from a Snowflake perspective. It's really brilliant. So that sort of data sharehouse concept will eliminate so much unnecessary data movement, to quote the Amazon guys, undifferentiated heavy lifting right there's no value there in moving that data it's just it's just heavy lifting so what let's remove that heavy lifting remove all of the monitoring process that had to watch this data integration and just query the data where it sits does that does that happen that often though really I mean are there many customers that want to share their data with other customers I mean I thought if

anything that's the last thing they want to do, really. I mean, that actually would be maybe introducing a concern into their mind that this data is really only partitioned by very thin kind of Chinese walls, really. I mean, is it something they've got? Are they getting take up of this or what? From my understanding, so we don't have any Snowflake customers that are actively doing that. But, you know, absolutely, I think people are thinking about this. Even Snowflake sells into departments a lot as well. So they're not always selling into IT. So it's not uncommon for one enterprise customer to have multiple Snowflake accounts.

So that becomes immediately valuable because now you I have? Because now you're not, you know, you can imagine a customer saying, well, let's think about Snowflake. But let's take a long sales cycle because we want to find out every department that might want to use Snowflake. And let's get everybody on board and let's create a Snowflake data warehouse. You don't have to do that. So departments can spin up Snowflake instances and share data between them. And I think, you know, I'm not a security expert, but this stuff's really secure. I mean, SSH keys and encryption and all of that, I don't think customers should be concerned about their data sitting in a cloud platform. No.

And I guess going back to the Paul Sonderiga episode of Drills of Detail, he talked about data monetization and exactly capital i mean this is yes it's a very forward-looking architecture really yeah absolutely i mean it is your it is some of the companies that you've been talking about on the show for the last couple years these are these are these are companies where the data is the product i think i may have said that actually on our first first time i came on show. And the data is the product. And the idea of that being your capital, I think where you work today, it's very much true that the data is the most important thing that the entire sort of platform sits on. Absolutely. Okay, so let's get on to building analytics at Google. So I bumped into you in London a while ago with a rather kind of

rather smug look on your face and your colleagues as you were actually working in Google doing some BI projects for them. So we had dinner and you talked to me about it and what you were doing. Just tell us a bit about that project at Google, who it was for and I suppose what was what interested you at the start about it before we get into the detail? Absolutely. So a quick shout out to the team that worked on it because I forgot to mention them when I recorded the Google Cloud podcast.

So Phil Gert, who was, and Emily Carlson were the two, and myself were the three people that worked on the project. So I got that out of the way. They did the work and you argued with them. Yeah, exactly. And I sat and watched. So it was for Google Play Marketing. So it's a team of actually Google Play Marketing communications.

So these are the folks that try to be helpful to Android developers or really anything that's being mostly Android developers. So they are trying to engage with these developers and make it easier for them to deploy Android-based applications. So that involves outreach to them through a variety of digital channels, some of them being public Google platforms, some of them being private Google platforms, and some of them being complete non-Google platforms at all. Those are the different avenues they use. So Twitter, LinkedIn, Medium, their internal email marketing platform, which is not a public Google thing. These were all some of the sources that they were looking at trying to ingest into a single place, as Dom Elliott, who was our main stakeholder from Google, would say

that all of these platforms have analytic capabilities, but they would be going between these different platforms all the time and pulling data and trying to put it together to give a unified view of user engagement and the health of the Android channel, really. So they decided that what they really liked to do is they had been looking at Data Studio and saw all the great sort of visualizations and capabilities that Data Studio

provided. They knew that instead of, you know, building these monthly data decks and Google Slides with all of this manually compiled data, that they could create a Data Studio document that had all of this in it. But they knew they had to get the data into a single place. So they chose the Google Cloud Platform mostly because they're not engineers. They're marketers, and they didn't necessarily have access to Google engineers. They probably could have found some, but it was much easier for them to simply really engage a company like ours that has expertise in the public cloud and to just do all of this running in the google public cloud yeah so so there's actually a podcast you recorded with uh with them which is out on uh

itunes as well at the moment we'll get we'll get the we'll get the kind of address of it later on and so on but i was listening to that and I was listening to that and again going green with envy while I was listening to it because I mean interestingly a lot of the tech a lot of the technologies you're working with and I'll get you to explain those in a second are things that we're using at Qubit. You also got to work with Google which is interesting and you built I suppose an entire I suppose data ingestion process using APIs and using things like PubSub and so on. So just tell us a bit about where did APIs come into it and what was the primary way in which you ingested data from those sources into BigQuery? So I'll start with the non-API stuff first and then get into the

API stuff. So we also used BigQuery Data Transfer Service, which is a sort of pre-built data integration platform. You just enable it in the browser with four or five clicks, where if you have data sitting in a supported Google platform, such as AdWords, I think it's AdWords, yeah, YouTube, other things, then you can just do a few clicks, specify your BigQuery data set where you want this to be replicated, specify a few things like table prefixes and whatnot, and it'll automatically start moving that data over, and then you can go back and also do historical loads, and you just sort of click a month at a time or a quarter at a time.

So we did utilize the data transfer service for the things that were low-hanging fruit, Google platforms. But then there was this whole series of other data sets that weren't in Google platforms, Twitter, LinkedIn, Medium, and we also did Google+, and investigated some others and had some others on the roadmap before the time ran out. So the API-driven approach was that all of these platforms have data readily available behind REST APIs. And so the primary way of getting data from these platforms is not the way we would have done it in the old days, which is, you know, think of this relationally. It's probably in a relational database of some kind. We want it to go to BigQuery, which is relational. So let's get some ETL tool that does relational to relational. That's not really how the modern world works. It's all API driven. So we considered

Dataflow, which is the Apache Beam SDKs running as a runner in Cloud Dataflow, the Google platform. But there's several things that didn't really make that ideal for us. One is that you don't write Dataflow in JavaScript. It's Java and Python, and that's not necessarily problematic, except there's no real pre-built connectors for streaming from REST and other things. There are some connectors planned and under development. But at this point, there was no real easy way for us to have used Dataflow. And also, since there weren't any Dataflow streaming capabilities for any of this anyway,

we would have either had to write those connectors or use something like Apache Airflow to try to orchestrate these sort of processes. And our driving principle for this project was to not introduce any amount of complexity that wasn't necessary because our customer had no engineering resources at the end of the day really to support whatever we built. So then we started looking at this very much from an API-driven approach, sort of the way that you might look at it if you were building an application, a front-end application, is that we wanted to go call these APIs, get data, and then build an event-driven workflow that processed those data ingestions and landed them in a couple of places. BigQuery was primarily where we were reporting on it, but we also did a data lake-like functionality by loading it to Google Cloud Storage as well. So the raw data was written to Google Cloud Storage.

So ingested, so App Engine is the application. We wrote JavaScript code in App Engine to actually go out to these different APIs and pull it. And then App Engine would ingest that stuff into PubSub, those messages. small bits of JavaScript that run in a multi-tenant serverless architecture that just respond to those data ingestions and do things. And so the whole pipeline is really a combination of App Engine at the front, ingesting data into PubSub and Cloud Functions, listening to PubSub with multiple subscriptions and doing different things with the data once it landed. Okay, so tell us what was it like developing in Google Data Studio? Because I've sort of been impressed with it in some respects.

I like the fact it's part of, I suppose, Google Apps, and you've got the whole kind of Google Drive integration. Exactly. But compared to what we're used to using, which is the ability to blend data from different sources and so on, it's quite primitive in some respects. Again, for anybody who's new to Google Data Studio, maybe just paint a picture of what it's like first of all but what was it like developing using it absolutely it's it's primarily a data visualization tool so

you're building visualizations on a click and drag palette it's sort of got a desktop publisher feel to it. And you build data connections and define those data connections. It's in a separate file. And then you can configure different SQL queries to go against data at sort of the page level. So yes, from a big box BI tool perspective with complicated metadata layers, even reactive metadata layers like Looker has, you might think this is a non-starter, but really it's so easy to transform data the way you need it and land it in BigQuery that we just you know sort of in the old school sort of pre BI tool days when you had to get the data into a model that was easily report easy to report against you sort of have to take that same approach with Data Studio because it's not going to mash up

data sets effectively for you. I mean, you can have multiple data sets on the same, they're technically called a report, but you can think of it like a dashboard page. But you can't blend data together in a single SQL query, for instance, the way we're kind of used to. So we knew that we were going to have to do a data transformation piece. And instead of using an ETL tool, we just did that in our event-driven architecture. We're having to blend streaming API messages with batch API messages. We're just doing all of that in Cloud Functions and writing BigQuery.

So we did curate the data set quite a bit um and sort of elevate uh you know create that that good old-fashioned model that's easy to report against and then data studio really really uh was very easy um with that sort of modeled data okay so so i actually have worked at google in the past and i was there with uh with michael rainey a few years ago and um and they're an interesting company to to work with i mean what was it like um what was it like dealing with google and what were they like as a customer and i suppose how applicable is their situation and their requirements to the other customers you deal with because surely it's on a massive scale and

completely different sets of users and requirements yeah i mean i think you have to kind of think about who our customer was so our customer was that was an int was simply some end users right i don't mean to minimize that but they weren't engineering folks they they they weren't um you know there weren't uh how do i say this there weren't, you know, there weren't, how do I say this? There weren't IT restrictions or constraints being imposed upon us. It was just simply what's in Google Cloud Platform. Anything that's in Google Cloud Platform is applicable. We obviously had to deal with certain teams and things about privileges and permissions to internal Google systems. But in general, the sort of the size of Google really didn't come into play because we were working with a small department within Google.

So from that, it was almost more like working with a startup, to be honest with you. They had visualization and data requirements. They said anything that can be done in the Google Cloud Platform is on the table. Now, we started removing certain technologies from scope immediately based on whether or not we thought the customer could support it once we left. That was a very big requirement for us, which was these weren't engineers. They needed to be able to support this once we left so whatever we built it needed to continue to run to scale to do all of those things after we left yeah so and it was in king's cross i think google around king's cross i mean that's that's that's changed quite over the years isn't it since you were there with me uh yeah absolutely so many years back it is uh quite posh now, to use the British term. It is gentrified.

Yeah, it was quite horrific when you and I were there before. I remember your face coming over from Atlanta. It was certainly an eye-opener, wasn't it? It was. The office is beautiful, and obviously it's got all the great things that any Google office has, which is tons of free food and lots of free coffee. So that was, we really enjoyed working there. I mean, we would look forward

to the next onsite trip we would have. Excellent. So just tell us where you can get details of the presentation you did on that and the podcast, and then we'll get onto our special mystery guest afterwards. Yeah, certainly. So I gave a presentation called what we learned building analytics for google i gave that at a utah big data tech conference uh you could probably at redpillanalytics.com there's probably links there i'm i'll provide you with some links more so that

you can put in the show notes um and then the and then podcast um um i'm thankful that you'll also put the link to that podcast it's the google cloud pod google cloud platform podcast which i was already a huge fan of and listened to every every week and was really honored to be able to uh to to have been on and been a guest so i appreciate you putting show notes uh and links in the show notes for that as well excellent so now i'm going to try and patch you in now so i'm going to try and patch our mystery mystery guest in um now so if it goes quiet it means it hasn't worked at all and i'll have to kind of have to carry on recording in a second but um we're trying i'm going to try and

patch them in and though this is a complete mystery to to you stewart as well so uh it's but you're going to i can't wait so so you're going to do the you're going to you're going to greet our guest and you're going to do the interview and I might chip in occasionally um but I thought it'd be interesting gink seeing you seeing as you were actually our first ever um guest to touch your hand over the reins for a little bit and let somebody else uh do the questions and uh we'll see how it goes so I'm just going to see if I can patch them in now I've actually done that to do this in Skype so let's uh let's try and do this now. And I'm about to send a gift to you there, actually.

So that's not it. OK, so let's try and do that now. Hello, hello. Oh, my goodness. Hey, Stuart. How are you doing, mate? Mr. Gorbachev, tear down that wall. Ugh, any time.

So, Stuart's going to do the interview now. So, Stuart, do you want to ask Alex to introduce himself, and I'll give you your questions, and off you go. Let me take over and see how the hosting jacket fits. So, with us is Mr. Alex Gorbachev from Pythian. Alex, do you want to go ahead and introduce yourself properly to anyone who hasn't listened to the podcast before? I don't think I've ever been on a Mark's podcast. Maybe that'll be the last time as well, depending how I do. So everyone, my name is Alex Gorbachev. I'm working at Pythian.

Been with the company for 11 years. Been doing everything from managing databases to building a new business in a new region. And being our CTO and incubating new practices such as big data, DevOps, and data science. And today, I'm actually heading our enterprise data science practice. So that's something exciting. That's fantastic. So when we saw each other down in Brazil earlier, or actually last year, you were doing something a little bit different. Is this new role, it's reasonably new, yes?

Yes and no. I started our big data practice more than four years ago, and as part of this, we started a data science practice about three years ago, which, you know, been developing since then. In the last couple of years, I've been more focused on Python internal digital transformations.

Right. You know, basically converting our systems to agile, modern software in a cloud. And, you know, I learned a thing or two about, you know, cloud architecture, software, agile software development and things like that. So that was interesting.

And now I took back our data science practice to apply what I learned in data science because it turned out that's a big gap. Yeah. Mark is miserable. He's got two agile advocates on the show at one time. He doesn't know what to do with himself. So I got a few prepared questions, Alex, that Mark has peppered me with.

I'll maybe modify them a bit as I go through. But yeah. Yeah, just the starter, yeah. Excellent. So what is your first memory of meeting me? Oh, dude. The problem is that the first time I've met Stuart might have been just a hard day to be memorable at all, you know, based on how I stayed there. Because I think it was at UKUG in Birmingham. I believe you're right.

Please help me. I believe you're right. Yeah, and it must have been at the pub by the canal, by the Birmingham conference center. Probably very late as well. Yeah. That's my memory, at least. And you were still very much the Oracle database expert at that time, the Oracle rack guy. Yeah, at that time we were more on Oracle technologies. Yes, that is correct. So that leads well into the next question,

which is, is Oracle still relevant? Well, yeah, it's a very big company and they have very many customers. Yeah, exactly. So by all means, it is still relevant. Okay, and so we'll drill into that just a little bit. So in what way are they relevant? Obviously, their database business still has a lot of customers, but the new cloud things they're doing and some of the new architectures and sort of trying to go after the Amazon and Google market for developers, are they still maintaining relevance or we'll say extending their relevance into some of those new avenues? Like, I think when it comes to, you know,

Oracle technology stack, you know, database and, you know, middleware, the strengths of Oracle and those technology stack, I think coming from two sides, you know, they have a very strong, solid, mature customer base. They also have quite a bit of money to spend as opposed to, let's say, like a startup, unreliable startup scene, you know, involving companies. So that's a big, big strength. Of course, they have a solid, mature technical product being a database. And, you know, the middleware server is not that bad either um one of the leading um java middleware servers right yeah um so and uh you

know they have some strengths in the bi space as well although that space you know is uh i think much more competitive uh but i think the challenge that they have is that although their technology is mature and very capable, and, you know, when it comes to relational databases, arguably the number one, unless you ask maybe somebody from Microsoft's side, but, you know. So I think the problem is that the technology that Oracle has and their business model especially is not quite aligned with the modern cloud native architectures and the trends like, you know, architect stuff out of microservices.

Make it commodity based. Make it scalable, like different than Oracle products normally scale and so on. And, you know, Oracle like adding some shims into their product space in order to make it look like that. But the foundation of that are quite different. Now, having said that, the vast majority of enterprise applications and where reasonably large is spending money on, so they still are full of those monolithic-type architected apps. So they're not really going to go anywhere easily anytime soon, right?

So re-architecting, we both know, is very hard and expensive. So generally, the way that this technology is phased out is that they're replaced with different types of applications. And oftentimes, it's SaaS apps. Yep. So as a result, Oracle's push into SaaS space is very understood. It's definitely the right way to go. And they also try to restructure and reinvent the technology space as well, which is traditional strength of Oracle.

Excellent. So we're going to shift into a different direction a little bit. So you and Mark were both two of the people that I recognize from the traditional BI, sorry, traditional Oracle space that really got out in front of Hadoop. And you were both proposing it and really evangelizing it very early on. Is it still relevant, though? Have cloud architectures made Hadoop no longer relevant? I guess it depends what you put in the name Hadoop, right? I mean, early on, from my perspective, when I look at it,

there's two major things that Hadoop was bringing on, you know, three to four years ago. On one hand, it brought an architecture which is scalable and affordable, or which allows to build affordably and scalably large-scale data systems right and in in many ways it was because of you know um hdfs uh and you know uh early on map reduce and later on some other distributed frameworks and that worked you know really well however at least the the the hdfs paradigm it wasn't quite designed for the public cloud infrastructure the way that we know it today, right? It was more designed on the way it was designed this way is to create a certain architectural decisions of collocating data and storage together

because it's really hard and expensive to separate them. And the public cloud vendors went different ways, right? Instead of doing that, they actually optimize their systems so that it's actually affordably to have a large-scale networking and storage throughput capacity to a certain degree, right? But for the most, you know, mere mortal applications, you don't need any more of this HDFS-like architectures because the commodity price cloud infrastructures are capable enough to process a lot of the workloads without this architecture and compromises that come with that. So that's what I think was relevant three, four years ago when it comes to Hadoop, which is something that's right now not very relevant.

Now, another part of Hadoop and why it was popular is that it's also created an ecosystem of like integrated tools and products that kind of worked really well together. And when you are early on in the kind of new space, new evolving space, having something to kind of unite around and integrate around was very important. Sorry to interrupt you. It's the value of the distribution, right? Exactly, right. And another few distribution vendors, CloudBerry, Hortonworks, MapR, came in and created distributions around those, then it becomes really, really easy to collect a variety of open source, mostly components,

and make them work together reasonably seamlessly, right? So that was really important. It's kind of still important, although the space has matured significantly these days and with the offerings from you know our cloud vendors those offerings themselves become very mature and you know overlapping a lot with what the system been bringing and you know thanks to proliferation of API and understanding how to design and create products that

some more micro services like that easily integrate and embeddable, this kind of become easier, or without, you know, the need for a Hadoop distribution of some kind of a collection of a software that runs seamlessly together, because cloud vendors started doing that. And fantastic answer allows us to do that. So that's kind of how I see Hadoop becoming less relevant today. Yeah, fantastic answer. And APIs allow us to do that. So that's kind of how I see Hadoop becoming less relevant today. Yeah, fantastic answer. So we're going to shift again.

And what's it like being in your current role at Pythian when your CEO is almost as well-known as you and perhaps crazier? That's my question as well. Yeah, that's Mark's question. Well, it depends where you look at it. It would be much more known than me for sure, or in more areas. I guess maybe some context. Some context for me. I mean, I guess, what's it like working in a place where it's very dynamic and your CEO

is very kind of forward looking and things change very often and your job is to keep everything running at the end of the day keep the trains running yeah uh i think uh i think first of all i think at this end there's much more people who are responsible for doing that other than me and paul another company grew bigger, they were significantly more. So I think, you know, maybe five years ago, I've been kind of playing a more prominent role, and I'm probably just being humble, possibly, but I think today there's a significantly more bright people, and Python is bigger itself.

That there's more than just, you know, me and Paul that are visionaries significantly more. If you look at our blog and you see who is there, if you look who's presenting at the conferences, which is not only in Oracle, Microsoft, and MySQL space these days, we've presented at big data conferences and DevOps conferences and et cetera.

So what is it like? I don't know if I'm doing it really well, right? I've been changing my jobs quite rapidly, right? Even as a CTO, I've been working on internal systems for some time, then I shifted to grow practices, and I shifted to do some internal transformations again. Now I shifted my focus as opposed to being a more global CTO of the company. I actually narrowed it down to a data science practice. Right. So, um, um, I think like from my personal perspective, I really like, I mean, as

you said, when the company dynamic and, you know, keeping, keeping up with the trends in the industry. And then that gives us ability to do what we want and focus where we want. Sometimes it's a, you know, a broader focus as a CTO. And sometimes in this case, I'm, you know, passionately building our data science products and service offerings. So the next series of questions, Alex, I'm going to ask you to choose between two technologies. You pick one of them, and you tell me why you would choose it. So first is Google Spanner versus Amazon Aurora. Well, that's a tough one, depending on our criteria, how we evaluate them.

I would pick Google Spanner for a more long term as a technology that's definitely been designed to be cloud native and uh from scratch based on you know very mature research that's been done so i feel long term the google spanner is a significantly more innovative approach i would pick aurora for many day-to-day tasks today as a more mature technology. More mature technology, more known to people who are used to MySQL. But Aurora is leveraging a lot of

and kind of constrained to some degree by MySQL as a kind of a framework that it's built on. So to some degree, that's kind of limiting to it. So it depends what you're asking for. I think long-term, my bet is Spanner as the innovation to follow and focus on. In the short term, I think Aurora is more usable. Yeah. Do you see an organization

maybe building a massive online transactional system using Spanner? Are those days ahead of us? Well, Google itself supposedly did that, right? I mean, it's still a reasonably young product, right? So is it out of beta yet even? It was, right, GA? It is. That's my understanding. Yeah, so I think it's still relatively new.

So I haven't worked on a large-scale Spanner deployment yet. Okay. So I couldn't comment on that. But Aurora has definitely been around more and been used much more and we've had many customers at Python who have been leveraging it. So, of course, proliferation of Aurora is bigger right now than Spanner.

Okay, so similar choice between the same two companies, but different technology. Google BigQuery or Amazon Athena? Good question. I think you would probably be better comparing. Well, let me put it this way.

The combination, so BigQuery versus the combination of Redshift, Spectrum, and Athena. Ah, now you made it harder to answer. Well, the Redshift is basically glorified Paracel technology on steroids, right? On the Postgres based on its own, right?

So again, if you just compare Redshift with something like BigQuery, again, long-term BigQuery, it's been redesigned from scratch it's a much more cloud native architecture it has completely customized execution and storage engine design just for you for each other and at the same time it also enables you know the most uh uh the way as athena runs or a red Spectrum, right? When you actually decouple storage from computer, which is interesting thing. Just like as you see what's been done in Hadoop, it's now found all its ways in a cloud, non-Hadoop technologies as well, right? Right.

As a side track from what we talked about before. My view on this is based on this. Amazon being, as I know, they developed some of the things from scratch. Amazon, I feel, is much more known from taking some of the existing products, specifically open source products, such as Presto for Athena or Postgres-based Paracel, which isn't itself open source for Athena or Postgres-based Paracel, which is in itself open source but in Postgres, and then taking this and base a product out of this. Now, I wish they would also actually contribute more of that stuff back to open source, which is a bit of a sore spot for me personally.

The way Google does, right? Yeah, I would much prefer the way Google does it. And although Google BigQuery isn't itself open source, right, Google kind of made it based on their own research and published a lot of that for us to use. So from that perspective, I think I would pick up a BigQuery just as much more community, I think, friendly and respectful option so next is the Google kubernetes engine versus ECS or the elastic container service in Amazon I

think kubernetes is winning the world on this yeah absolutely I mean that that that one's been solved hasn't it yeah yeah and even Amazon is not supporting kubernetes how it is it's true you mentioned uh uh google cloud functions earlier on and obviously there's lambda functions in aws and so on i mean to both of you really i mean is that the next a serverless kind of functions and so on is that the next thing people are going to be using to build um other applications event-driven things in the cloud? You want to go first, Alex? Yeah, I strongly believe that this is the next move after containerization, actually.

I don't know if it's in parallel with it. I think it's more high level of abstraction, how you run your code than just containers. So I think it may need to be more broad to kind of completely win over containers and make containers irrelevant. But I definitely think it's more of a bleeding edge compared to just containers. And in this case, it's interesting because AWS seems to be somewhat ahead of Google when it comes to serverless functions. And Google is kind of, is a little, I feel like, behind with their cloud functions, the way that they adopted and the functionality of them on GCP, and much more pushing towards

containers and take leadership in container space. and the functionality of them on GCP and much more pushing towards containers and data leadership in container space. Yeah, Lambda still, you know, Lambda gives you the ability to actually schedule functions. So the problem that Google has with Cloud Functions is you still need something else to sort of kick off the event-driven architecture. You can't write the complete thing.

So, for instance, the project we discussed earlier, Mark, that's the main reason we had App Engine, is that Cloud Functions couldn't itself initiate the workflows. So Lambda certainly has the ability for you to schedule functions. And so you can write your code, I would argue, completely, depending on what you're trying to deliver, completely within one platform using Lambda. So the next question, Alex, and it is the last one, and obviously,

it's a good one based on your earlier responses, which is, how do you keep yourself so current? I mean, obviously, you've got a broad command of traditional technologies. Most people who know you for a long time know that, but also you're well-versed in modern technologies, and you seem to be right there on the bleeding edge. How do you do it? Well, I think that's a lesson I learned from the number one, Paul Vallée, and this is why he is the number one. One of the strategies he taught me is to, you need to surround yourself with a brilliant people who are mostly better than you at most things. And part of that is that you end up learning from them a lot because you look like a stupid idiot if you don't.

It's like the no bozo factor at Apple. Exactly. You got to do it. Otherwise otherwise you know but uh yeah i mean it's a lot of learning from the peers of course it's quite a bit of reading and trying things sometimes of course it's impossible to try you know a lot of things yourself these days and stay current in a broad area so sometimes like you make you have to like reasonably be reasonably shallow on certain things. But have a big picture in mind and the ability to get your hands dirty and go do some coding. Just before you conference me in, Mark, I've just been coding our coding our psychic learn new psychic pipeline with one of our data scientists here so i keep my hands dirty at the same time so it's excellent ask questions

you as well i mean you've been you've been i mean you and i've been doing consult what you've been doing consulting now especially the last couple of years how do you still motivate yourself i mean it's it's you know you're running a business you're keeping yourself i mean before we're joking you know you you you know you are pretty on the ball when it comes to technical stuff. And you're running a business. How do you keep yourself kind of motivated and current as well? It's really easy. If I was an insurance salesman, this would be my hobby.

I think this is my hobby. It's the thing I'm most interested in. And so when it comes time, when I have a little bit of downtime, I'm not actively doing business, running the business or whatever, and I have a little bit of time, I'm dipping my a Docker container and seeing how some new piece of technology works. So it really is that it's my hobby. I'm lucky that my hobby is actually how I earn a paycheck. Yeah, exactly. I think it's the same for all of us, really. I think if you're fundamentally interested in it, it fundamentally makes you get up in the morning thinking this is really exciting. It's good. And I think the industry's changed a lot, even just the time that we've been working with it i mean the days of big releases every kind of year the days of uh of spending

most of our consulting projects actually installing the thing and not actually kind of getting value from it it's changed as well but i suppose the fundamental thing of how to solve a problem how to you know apply technology and use cases and and uh and those trade-offs as well between you know what you said earlier on you know there's lots of technologies you could have used at Google, but you didn't because you wanted to make it sustainable. You know, that's, I guess, why you're still in business, both of you, sort of now. Well, thanks very much for that. It was really a pleasure coming on.

And Alex, you were a pleasant surprise, my friend. Well, it's been a nice surprise too. I'm not doing too many of those these days, so it's fun. I told him it was Don Burleson, first of all. So I think he was quite pleased when it was you, Alex. Although it would have been interesting if it wasn't. So thanks to both of you. Thanks, Stuart, for being a good sport as well.

You genuinely had no idea who was coming on as the guest. And you've done brilliantly interviewing Alex. And Alex, thank you very much for coming on as well. it's been good to speak to you and uh i do remember the first time you two met actually and i'm not surprised you can't remember it because it was uh i think it was it was uh it was it was stewart's first introduction to the tap and spa and uh and uh the british drinking culture and uh the next morning i think about nine o'clock which was uh you gave gaming up the next day wearing a suit and look like you've been hit by a bus the night before so i remembered remembered it corrected that exactly you did exactly so

thanks very much both of you and uh take care and uh yeah speak soon yeah thank you mark and thanks steward A multi-tenant cloud-native platform where all customers date. My microphone just spontaneously fell over then, actually. Sorry. It must be the quality of my jokes. Right. So carry on.

I'm going to edit that bit out there in a second. So you were just saying about something. So carry on the bit you were saying. Yes, sure. And I'll edit it back in again. Yeah. you were just saying about something. So carry on the bit you were saying. Yes, sure. And I'll let you back in again. Yeah, so in the old days, so what Snowflake realized, I'm going to start one more time, Mark.

Yes. Think about Data Shithouse as well. Yeah, exactly. I just thought that, I was reading it, I thought it was quite good actually. All right. Excellent.

So for the next question, I'm going to ask you a couple of things. I just want a yes or no answer for about the next three questions. Sorry, not a yes or no, but an A or B. It's multiple choice here. Okay. So Spanner or Aurora? You have to give the context, but you know, okay. Amazon Spanner or Amazon Aurora? Spanner. Okay. Google Spanner versus Amazon Aurora. you know okay spanner amazon spanner or amazon aurora spanner okay context as well but all right yeah you want them to provide yeah let's stop a second right let's do that again right so what i meant with that was was actually to put that to i say a or b but then

give us some reasons behind it yeah so so and I think it's an opportunity for you two to discuss those technologies a little bit, really. So. You got it. I'll restart then. Go then. Let me restart with a question. Just to say, Stuart's had absolutely no preparation time for this and had no idea it was you coming

on the call. So he's doing pretty good so far, actually. Go on, Stuart. So we're going to talk about a couple of specific technologies here next, Alex. And I want you to weigh in on, if I give you an option, an A or B option, you pick one of them and tell us a little reason why, okay? So first, Amazon Spanner or Amazon Aurora? Stuart, Stuart, Stuart, it's Google Spanner.

Oh, yeah, you're right. Sorry. Do it again, do it again. I'm going to start again. You sound like an idiot otherwise, so go and do it again. Thanks're right, sorry. Do it again, do it again. I'm going to start again. You sound like an idiot otherwise, so go and do it again. Thanks for that, Mark. So Alex, we're going to shift a little bit here. He's not normal then.

Sorry, couldn't resist. I'm now into my normal way of thinking, right? Idiot mode. Okay, so restart.