Orchestrate all the Things - Open source observability, meet data transformation: Grafana 7.0 promises to connect, unify, and visualize all your data. Featuring Grafana Labs CEO and Founder Raj Dutt

Episode Date: May 29, 2020

Grafana Labs, makers of popular open-source observability platform Grafana, announced the general availability of Grafana 7.0. This comes only a few months after Grafana Labs scored $24 million in Series A funding to double down on open-source strategy and build what it dubs the world's first open and composable observability platform. In this backstage chat, CEO and Founder Raj Dutt and George Anadiotis connected to discuss Grafana 7.0, and the road forward. Article published on ZDNet, May 2020.

Welcome to the Orchestrate All the Things podcast. I'm George Amatiotis and we'll be connecting the dots together. This is episode 2 of the podcast featuring Grafana Labs CEO and co-founder Raj Dutt. Grafana Labs, makers of popular open-source observability platform Grafana, announced the general availability of Grafana 7 in May 2020. Raj and myself connected to discuss the new release and touched upon an array of issues. Open source, data integration, processing and analytics,

and growth in the cloud and on-premise. I hope you will enjoy the podcast. If you like my work, you can follow link data registration. No worries at all. Thanks for taking the time, George. Appreciate it. Okay, great. So yeah, I was just telling Kathy that I spent the last few minutes

at least going through our previous discussion and a few topics that we left kind of pending, let's say, and of course your press release. And it seems like I was just commenting to Cathy that it seems like the path you laid out you have been kind of executing on that so I'll just let you pick up from there I guess. Yeah I think so we last talked in I guess it was October right after our fundraiser right George?

Yes exactly. Yeah. Cool. So yeah, lots has happened since then. I guess it's been about eight months. And yeah, I mean, there's so much to report. So I'll just dive in with some updates. So, you know, when we last talked, we just, you know, kind of hired, made some key hires on our go-to-market. We were scaling up sales. Obviously, we had just raised the $24 million. And we were all blissfully aware of the

clouds on the horizon with regards to the global pandemic. Right? Well, too bad you didn't let the rest of us in. Yeah, right. So, one let the rest of us in. Yeah, right, right. So one of the most exciting things that I'll start with is on the product side. When we last talked, I think you'd written about open source observability as combining metrics, logs, as well as traces.

And we were planning on adding tracing support to Grafana. So one of the big news items that we'd like to go over is on Monday, and I think you've got the press release under embargo. I do. Monday, and I think you've got the press release under embargo. Yep, but literally Monday of next week. So, you know, three days from now, we're going to be announcing Grafana 7. And one of the major features in Grafana 7 is we add support for tracing. And so that kind of completes our observability story in a way, because now Grafana can visualize and now transform, which I'll tell you more about, metrics, logs, and tracing. And we add support for two of the most popular open source tracing databases or tracing platforms, Jaeger and Zipkin. So Grafana

ships out of the box with support for those two platforms on the tracing side and will be continuing to add new tracing integrations over Grafana 7X, including APM companies like Derelict, LightStep, AppDynamics, Dynatrace. And this is in the same Grafana way where we really prioritize interoperability with other tools and allow our customers and users to set their own observability story rather than being prescriptive and requiring them to store it all in our database. So tracing support is brand new in Grafana 7.

We're really excited about that. There's a couple is brand new in Grafana 7. We're really excited about that. There's a couple of other features in Grafana 7 that we're excited about. I mean, there's literally dozens and dozens of new features. We don't have the time to go over them all, but two other features that's mentioned in the press release are support for transforming data, not just visualizing data.

So what that means is before Grafana was all about visualizing the data that you queried from all its different data sources. And you were kind of limited in a way because if you couldn't kind of massage the data through the query language of the underlying data source, you were kind of out of luck. You had to visualize really what you queried. And there was always very flexible ways of visualizing this data, but ultimately you had to visualize the data that you queried. So now the story is there's an addition to the visualization pipeline. So instead of just querying data and visualizing it, you query data and you can actually transform that data

before you visualize it. So you can do joins, you can do calculations, you can do reduced jobs. And what this means is the power of Grafana has really been extended. And it's not just about combining different data to visualize it, but you can massage the data to get it in the exact form that you want, even if the underlying data sources don't allow you to do that.

I would say, yeah, that's a pretty big deal. And in a way, it reminds me of the kind of journey that I've seen streaming platforms take. So they went from straight ingesting, basically, to adding processing, transformation, and operators, and all these kinds of things. So I wonder, well, you mentioned your query language. So one part of achieving that would be to extend the query language with the right operators to do that. But of course, you also have to have the implementation to support that. So I wonder if you bought any pages, let's say, by how streaming platforms have done that. In some ways we have, and in some ways I guess we haven't. So essentially there's been a complete, and this is another point that's new in Grafana 7, but there's been a complete refactoring of the underlying data structures in Grafana.

So before the data structures internal to Grafana were really specific to series data, like metrics data and log data. And it was very oriented towards time series data, which is the roots for Grafana. So we've actually kind of been inspired by the Apache Arrow project. And now the internal data representations within Rufana are data frames. So essentially, that really allows us to have first class support for a much wider variety of data types, if you will. So things like columnar data or tabular data is really now first-class citizens. And we've used the Apache Arrow data frame format to do that. As far as query languages go, we've really kind of tried to avoid a very complicated or developer-focused experience and really

gone with an experience that's all click and driven from the GUI. So you can really add these transformations from any panel. different transformation types, like calculations, you know, like joins, you know, reducers, and you can really just through a point-and-click experience build your transformations up. You know, things like rename fields, do math across queries, you know, join separate time series or logs together. And you can basically create a pipeline

of these transformations and control the execution order of the transformations. And it's all a point-and-click experience from within the new panel editor You know also add labels to fields you can You know Kind of reduce fields to like maximum minimum Yeah, the way you describe it it sounds like I would say you know like a mini

Programming environment where people can you know, like a mini programming environment where people can, you know, join different data sources, transformations and conditions. You talked about reduced jobs and so on. And basically, you know, it sounds like a visual graphical user interface for processing data. Exactly. Yep, exactly. And so this data processing pipeline is brand new. And especially if you've got a data source that doesn't have its own, you know, rich, powerful query language, like, say, Prometheus,

and you've got a custom data source or your data is in a, you know... Google file or... Exactly, yeah. Then finally you have some basic things that you can do with the data, even if your database doesn't have any real query language. You know?

Yep. Okay, so two questions there. I mean, sure, there's no question that, you know, having a graphical user interface like this is super, super valuable. So question number one is whether this is part of the open source offering or the enterprise version. And question number two, I guess what probably happens when people operate this user interface is that it somehow interprets what people are doing on that level to the underlying query language. So my question is whether it's possible

for people who are advanced, if they want to just do whatever transformation they want directly operating on the query language or potentially intervene in the auto-generated code and fix it or optimize it or do whatever you want with that? Sure. So the transformations feature is absolutely in the open source version of Grafana. So the answer to that is yes.

And in Grafana 7.0, we don't expose the query language, but that's something that we'll be thinking about doing in later versions of Grafana over 7.x. We really wanted to focus on getting the GUI experience right, and there's still some open questions in terms of how we're going to expose this, but that's something we're definitely thinking about for 7.x. Okay, yeah, it makes sense. I mean, to be honest with you,

that's where I would start as well. And then, yeah, you can make, you know, for power users, let's say, you can make that available at the later point. Yeah, yeah, yeah. We do provide like a debug view where you can kind of see the inputs for your transformations,

the outputs of how the data is being transformed, but we don't yet provide a raw query interface for doing these transformations. Okay. And then the third most interesting thing,

and I'll stop at three top features with Vana7, and you can take a look at our press release and get more. But the third one I'm really excited about is improvements to our work. So if you'll remember during our last conversation, we talked about one of the main advantages of Grafana was we have dozens and dozens, I think there's over 60 at this point, of plugins that are created not just by Grafana Labs, but by the community.

And these plugins allow Grafana to have different data sources, have different visualization types. And that's, in many ways, what gives Grafana its superpowers. And this plugin framework was always pretty complicated for developers to implement new plugins, mainly because it gave them a blank slate. So it was pretty flexible, but it was also pretty daunting and sometimes pretty difficult to create a new plugin. So the plugin architecture has been really reimagined in 7.0. And what used to take, let's say, 1,000 lines of code to implement a particular plugin now only takes about 100 lines of code. So we really wanted to lower the bar for how easy it is to create plugins and not just how easy it is to create plugins, but also how easy it is to maintain those plugins.

Because Grafana is evolving and developed. So we think we've achieved that with the new plugin architecture. And of course, just by virtue of creating a data source plugin, you get certain Grafana features like alerting or transformations available automatically. So the underlying plugin API has undergone a massive reboot and it's much more developer friendly. Okay. Okay. Yeah, I will have to agree that yes, on both fronts, actually. I mean, yes. Any open source platform actually gains a lot by this kind of contribution from the community, so plugins.

So yes, it does make sense to make it easier for people to contribute, to lower the bar. What I wonder is, because the way you described it, it sounds like a pretty massive and impressive, I would say, transformation. So going from one scale of money to from 1,000 lines of code to 100 lines of code. So to be honest with you, I don't know something which is pretty basic.

So what's your programming framework, if there's a specific programming language that you're targeting, or do people work with Southern Array APIs? Or how does it work exactly? Sure. Go ahead. Sorry. And the second part of my question was going to be, how exactly did you refactor your API

to make this possible? Sure. So the first answer to your question is, it's now possible to create plugins through the API and through following our data format in any language that you want. So before, plugins were kind of front-end only, which means they had to be written in JavaScript. And now plugins are back-end plugins, which means that you have a choice to write it in whatever language you want. You can write it in Go or JavaScript like the rest of Grafana. You could also use

C or Python or Java if you wanted to. These backend plugins are actually executed by Grafana and as long as you're using the correct APIs and data interchange formats, you can really write those plugins

in whatever language you want and you're comfortable with. So a lot of, you know, we prioritize that because we have a very diverse community. And, you know, certain languages are more suitable for other things depending on the plugin you're creating,

depending on what language has good, say, SDKs with a particular data source. So it's very flexible now. You can write plugins in any language you want. Okay. Okay. Well, that kind of brings up a follow-up question. So because that's, I would say, that's a pretty interesting feature.

I mean, being able to host and execute plugins in whatever programming language. So how do you do that? What's your plugin execution environment like? So basically, the plugin execution environment for IAM plugins basically allows, so we just run the plugin as a process. And so, you know, it's as long as, you know, basically it's, the plugin is almost run like an executable. And,, as long as that executable follows the right data interchange formats and APIs, Grafana doesn't care what language it's written in because now it's in the backend. Okay. So it sounds like you have a kind of customized virtual machine, I would say, that's

from the side. Similar to that, yeah. It's not quite a virtual machine, but it allows you to run plugins in isolation as essentially binaries. Okay. Is it container-based by any chance? No, I'm not actually sure of the answer to that, George. Okay.

I was just curious. It's not all that important, uh um no i'm not actually sure of the the answer to that george um yeah okay it's not all that important but it sounded you know it started sounding a lot like a container so that's why i was wondering yep yeah okay for sure um so those are three of the the top things that we're really interested in and excited about on kafana 7. The tracing support, the transformations, and the new plugin architecture. And we had a look at the press release for some other interesting things. Yeah, I did actually. I have to say it's really helpful to have your view

because you're right, there's quite a few features. So yeah, it's good to know which ones are your favorites. Yep. And then on the company side and the, you know, sort of metric side, you know, Grafana Labs has really, you know, exploded in many ways since our last conversation. So, you know, over the last six months, we've grown the company from, I think when we talked, we were maybe, I think around 80 people or something like that. And today we're 140 people in 25 different countries. Obviously, we were always a remote first kind of fully distributed company. And, you know, so luckily, you Luckily, with the current situation, other than not

being able to get people together for events and meetups and things like that, our day-to-day operations really hasn't changed. Because even before the pandemic, our culture was such that we never required anyone to come into the office for any reason. So we're a remote-first company. We're now 140 people, 25 different countries. Our go-to-market investments are really paying off. The company of grown its revenue significantly since we last spoke. And so we're really seeing a nice growth curve there. And the growth curve applies not just to our revenue, but also to the adoption of the Grafana project. So at this point, we're approaching 600,000 active companies using Grafana every day. And that's up from, I think, about 300,000 or 400,000 when we last spoke.

So probably by the time you write your article in the next week or two, we'll hit 600,000. We're well over 500,000 as of the last few weeks. Okay. So that's an important clarification because I thought you said 600, but it's actually 6,000. No, no, 600,000. Sorry, I misspoke. Yep, 600,000 companies.

We're approaching 600,000 companies using Grafana every day. Today the number is over 550,000. Yeah. Okay. One of the things we touched upon the previous time was, well, the kind of classic by now dilemma, let's say, on-premises versus cloud. And you were very clear that going cloud was kind of priority for you and you were seeing growth in that area. So I was wondering if you have any insight as to how much of the growth that you refer to is coming from your cloud offering

versus the enterprise on-premise. Sure. So I'd say it's about almost 50-50. We're constantly surprised, like from a number of new customers or a number of new logos standpoint,

you know, it's definitely more biased towards cloud. But there are some very, very large deals that we've closed that are for Grafana Enterprise, like call it,

you know, seven-figure deals. I think that this is an interesting point because even though we think that the future is Grafana Cloud, I think people in general really underestimate... Of course, the future deployment model is cloud and that's what we're betting our long-term strategy on. But in the short term and the midterm, I think that people often underestimate the demand from the really large Fortune 500 companies who still want to have either a hybrid deployment or they want to have prem deployment. And the reason why they want to have an on-prem deployment is not just perceived security or compliance reasons. It's because their data is both on-prem and in the cloud. Therefore, they want their Grafana to be on-prem because then they can connect all their data together. If their

Grafana was in the cloud, it's very difficult because then they can connect all their data together. If their Grafana was in the cloud, it's very difficult because then they have to open up their firewall or poke those ACLs back into their infrastructure. It's easier to go the other way. If you want to connect your on-prem data with your cloud data, it's easier to run Grafana on-prem

than it is to run it in the cloud. So a lot of these companies have large Splunk installations or large SQL installations. And so as long as you've still got some of your data on-prem, the on-prem Grafana Enterprise offering is really appealing to some large customers. So to answer your question, we've seen almost an even split between Rafaana Cloud and Rafaana Enterprise. And in many ways, it's a two-horse race for us internally. It sounds to me from the way you described it that maybe in terms of number of clients,

let's say, or at least new clients, you got more for the cloud offering. But in terms of volume of how much your new deals are worth, maybe you got a few on-prem clients, but you'll be glad. Yes, exactly. The average customer size for the on-prem Rufana enterprise offering is definitely much higher than the average customer for Grafana Cloud. Because our Grafana Cloud offering, we just launched a free plan. But even without the free plan that we just launched this week, the entry price point for Grafana Cloud is $49 per month. Whereas the entry point for Grafana Enterprise is two orders of magnitude higher.

It's significantly higher than that to purchase Grafana Enterprise. I would say that possibly another reason for really, really big organizations going for either completely on-prem or hybrid developments is that they may have capacity on their own, like data centers of their own that they can utilize. So, they choose to do it that way. Yep. And to be clear, and I think you already get this, George, but when we say people buy Grafana Enterprise to run it on-prem, in many

cases, their on-prem is on AWS or on GCP or Azure anyway, right? They just want to control that themselves. Yep. I hope you enjoyed the podcast. If you like my work, you can follow Link Data Orchestration

on Twitter, LinkedIn, and Facebook.