The Changelog: Software Development, Open Source - Go in the Modern Enterprise and Go Kit (Interview)

Episode Date: July 4, 2015

Peter Bourgon joined the show to talk about building microservices using Go in the modern enterprise and his microservices toolkit Go kit.

Welcome back everyone. This is the Change Log and I'm your host Adam Stachowiak. This is episode 163 and on today's show we got Peter Bragon joining us to talk about Go in the modern enterprise. And today we're talking about GoKit kit a go toolkit for microservices great call today with peter and uh speaking of peter he's going to be at gopher con along with us uh and so many other people 1500 people are going to be at gopher con 2015 so we're going to be there with cameras in hand covering everything we possibly can with gopher con so say hello if you're there hit us up on twitter whatever it takes we got three awesome sponsors for this show code ship digital ocean and top towel our first sponsor is code ship a hosted continuous delivery

service focusing on speed security and, and customizability. You can set up continuous integration in your application today in a matter of seconds and automatically deploy your code when your tests have passed. CodeShip supports your GitHub and your Bitbucket projects, and you can get started today totally and absolutely for free. Should you decide to go with a premium plan, you can save 20% off any plan you choose for three months by using our special code. It's the changelog podcast. Use that code and save 20% off any plan you choose for three months. Head to codeship.com slash the changelog to get started.

And now onto the show. All right, everybody, we're back. We got Peter Bagan with us. He's a software engineer living in Berlin, Germany. It's 10 at night over there, by the way. It's 3 p.m. on my time across this pond, but he focuses on large-scale distributed systems. Peter, welcome to the show. Thank you very much. Thanks for having me. So, Peter, the conversation we're going to have is based around GoKit, but there's lots of other separate conversations that happen to describe why and what GoKit is and why you're doing it. Before we do all that, for those who may not know who you are,

can you describe what you do and what you work on sort of maybe some of your background oh sure um so i got my start actually in the telecom industry i was doing embedded software like cnc plus plus on i guess it was uh like routers and switches that you would normally stick into telecom switching centers at sort of the national level. So I kind of grew up as an embedded networking guy. And then over some time, moved into more distributed systems at a sort of higher application level. I worked at Bloomberg for a while.

I did a search product there, sort of a federated search. And I worked for some NGOs for a little while. I did a search product there, sort of a federated search. And I worked for some NGOs for a little while. I worked at another telecom company doing a monitoring product, sort of global scale monitoring product, and wound up at SoundCloud here in Berlin. And at SoundCloud, I worked on quite a few things. The search product there, I rebuilt that. I moved into the stream activities feed kind of area, which I guess is something that a lot of web companies are doing in one way or another. I got into distributed systems theory there, and I did a couple of projects related to that. that got the most attention in the world is a distributed sort of quasi time series database system called Roshi, which is based on CRDTs, which is a fun distributed systems data type, conflict-free replicated data types. It's all the rage. It's the hipster data type and distributed

systems theory. And then after that, I, and currently i'm working for weaveworks which is a company that does software defined networking for the container space so for docker containers so we uh currently are giving away this product that can network docker containers up together in you know the easiest way possible and then we're building building out stuff around that, basically. Very interesting. I mean, speaking of Docker and whatnot, DockerCon just happened. They just had a huge announcement about the app container spec

becoming, I guess, federated is maybe a way to say it. Yeah, the two houses combined. Yeah, I mean, that's good news for the users. so that must be some good news for weave as well yeah definitely we're is it we've works uh we've works as a company weave is the product of one of the products yeah i was looking at the page i'm like i just said weave and i don't want to mispronounce it i'm like okay is that right or wrong yeah this this seems really interesting interesting. So you've definitely had, you know, the full gamut of interesting software development experience embedded. I mean, doing something on just the device itself, how different is that than what you're used to now, which is distributed and,

you know, multi-layered services? How much different is it to just sort of embed software onto a network, you know, network hardware or something like that? Yeah, I mean, there's similarities and differences. A lot of the theory applies pretty directly, actually, no matter what scale you're working at. I would say the major differences are the sorts of things you have to care about. When you're focused on something so small, your scope is narrowed, I would say. So you spend a lot of time counting all the bytes and counting all the ops. And when you're working at a broader level, that stuff sort of fades into

the background and you're more concerned with correctness, especially in the face of failure. I think probably the biggest shift for me in my career is that I've become more and more concerned with building correct things quickly or rather than sort of eking out the last little bit of performance because that seems to be what's important or perhaps where the industry is going like we can buy servers to scale horizontally but if the systems are fundamentally broken, no amount of servers is going to fix that. So that's kind of been my experience. That's very true. So I guess the meat of this conversation, as we'd mentioned a little earlier, was really routed around GoKit. And this was originally a

talk that you gave at FOSDEM at the Google campus there in London at a meetup earlier this year. So back in February. So this is relatively a new idea or at least publicly a new idea? Yeah, yes and no. What happened there was Andrew Durand, who I know through the Go community, reached out to me and asked me if I might have anything to say at FOSDEM since I'm not too far away. And I've spoken at Go events in the past. And he kind of left it up to me what I should say or what I wanted to talk about. And I kind of did a survey of my mind and the state of the sort of Go community at the time.

And at the time I was working at SoundCloud and we had just sort of, or we were maybe in the middle of this period of change. Maybe the story will resonate with a lot of people working in organizations where when we were kind of young and excited and full of optimism and opportunity, we had a lot of languages that different teams at the company were free to implement things in. And that was great. It was like a Cambrian explosion. And we were very productive and very happy. But we were small also. We were maybe 20, 50, up to 75 engineers or something when that was still going on.

And so obviously that carries a cost as you get bigger. Support costs go up. And there's a natural tendency to kind of isolate to or like pare down to just a few sort of core supported languages. For example, at Google, I believe the supported core languages are C++, Java, Python, and now Go. But that was, you know, the most recent addition. So that's the choices that you have when you want to write something there and so similarly at soundcloud we started saying you know we can't really be putting haskell uh services into production forever um let's think about uh where our strengths lie let's think about the sorts of things we want to be supporting long term. And it came down to a few final candidates. We had a lot of people who were into Scala, which was great. A lot of services

in Scala. We had a lot of people into Go. We had built a lot of stuff in Go and it was running, you know, really quite well. There was a little bit of Clojure. There was a little bit of Java. There was quite a lot of Ruby as these things go. And those are kind of the languages that we settled on. But that process continued. And what I found was that the Scala camp was kind of winning and more and more new things were being written in Scala or Scala was being chosen for more and more new projects where, in my opinion, Go would have been an equally good or perhaps even better choice. So I started reflecting on why this was. And what I kind of came up with was that product managers and owners, people who aren't invested in the long-term success of projects and teams, they wanted some sort of sense of security or support from a

language, from a framework, from the technology decisions that they were investing in for a long time, right? Like a year, two years, maybe indefinite sort of time horizon. And while Go was like proving itself technically, it wasn't mature enough and it didn't quite have the library support that we needed in our architecture. So it was an easier or maybe safer choice to pick Scala, even though it might have performance characteristics that weren't as good as they could have been or might have a cost of complexity that was higher than it needed to be. So that got me thinking, and it made me realize maybe something that Go is missing is this collection of higher order idioms and tools. And I hesitate to use the word framework, but maybe something like a framework that can be used in these sort of, well, I call it a modern enterprise, these sort of organizations like SoundCloud,

is to give these middle managers and technical leaders confidence in choosing Go as sort of a long-term language for their application layer, for their business logic. And so I sort of rolled that idea around in my head a little bit and decided that would probably make a pretty good talk. And yeah, GoKit was born. break it down into all these different components that may or may not fit into this blessed framework that somebody might use sort of piecemeal. Right. It's sort of choose your own, choose your own things you need for your infrastructure, your architecture. Exactly. One of the fundamental ideas behind the Go Kit is that we're coming in. We expect to be coming into an organization that already is a little bit big, successful. They have some inertia. They have some momentum. They have some existing infrastructure that they're not going to

just throw away overnight. We want to kind of slide in where Go makes sense and we want to work with the stuff that's already there. So yeah, we're not a framework that you have to buy into 100% and you can only talk to other go kit services or something like that. We really want to be good neighbors and we want to be able to have a story that you can tell to your boss or to your engineering director. That's like, look, we're just going to use a little bit right now. If it makes sense, we're going to there's there's a future there. And I just want to make that future kind of look bright exactly this also seems like a you know not so much a fist fight but definitely a uh an attempt a try a fight to for go to win

over java or scala or other options in the enterprise you mentioned closure ruby uh and yeah java so it seems like this is your attempt to, because you seem like a Go champion, and you want to use Go where you can use it and not let Scala or others win, I guess, win the battle for the next project. The battle for Mindshare, yeah. Yeah.

I'm definitely a Go advocate, and that's a personal thing that uh was immediately apparent to me when the initial release happened and um the more energy i've put into go the more i've gotten out of it it really just aligns with my personal philosophies and my um preferences when it comes to programming but i want to be and of course this is going to sound a little political but i but i'm really being honest when I say it's not for me a fight at all. I don't want to see Go beat Java or Scala or any of these things. All I want to do is make Go a viable option for people who want to use it.

And if there are Go champions at your organization who think that they can be productive in Go, who think that there are a set of services that would work really well with Go. I want to lower barriers to adoption from people who maybe don't know a lot about the language or would tend to stick with safer choices like Java or the JVM. So yeah, it's really not about winning so much as just getting Go to the same playing field, I would say. So for those, you mentioned middle managers, product managers that wanted security and support from the language, and an advocate for Go, what have you learned about Go in terms of security and support for the language that's changed that would change those people's minds? So a lot of it, I think, is purely a matter of time. I mean, Go has only been in the wild for, I think, five years. I think we're coming up on a five-year anniversary, if I'm not mistaken.

Maybe six years now. Yeah, 2009. I just had Andrew Duran on the show not long ago, and I think we were talking about the five years then, because to rewind way back, I think episode, if I'm correct, I keep missing the number, but episode three, I believe, of the changelog, we had Rob Pike on to talk about Go, and that was November 27, 2009. Yeah, exactly.

So it wasn't long before that that Go was born. Exactly. That sounds true to me. So yeah, five and a half years or something, which to someone who's been in the language like 100% basically from the beginning, that seems like a long time. But of course, to somebody who is responsible for the engineering department of a business, that's almost no time at all. So I totally understand that they are a bit perhaps cautious about investing in something so young on a broad timescale. And I think that's actually totally rational. I think it's just a

matter of time for a lot of people. And they need to see not just one or two successful projects that, you know, super techie companies, but, you know, a series of successful projects and companies that are not so techie, that are consumer oriented, that are maybe B2B companies who can tell a good like business success story from choosing go rather than just a technical success story. And so I want to help the people who are capable of becoming those good stories, becoming Go advocates from a perhaps less technical and more business-oriented kind of domain. So that's like my impetus here. Well, let's dive deep into GoKit then. Let's figure out what it is, why you built it, and I guess what problems it aims to solve for the, in quotes, modern enterprise. Yeah, exactly. So the most important piece of context to kind of make transparent is that I'm designing this explicitly for companies who have chosen to go with the so-called microservice architecture or service-oriented architecture. And in my experience,

and I think correctly, that is something that a company or an organization should do only once they reach a certain size. So we were talking about containers a little bit earlier. And often, I think containers and microservices end up in the same bag of tricks. They end up on the same like Hacker News homepage. And they are related, but I think, and I feel they serve orthogonal kind of ends. They do different things. Containers are a solution for a technical problem of continuous integration, of continuous deployment, of like getting your actual code onto machines and running it in a predictable and reproducible way and they're great for that right whether it's whether it's app c open container format docker

like they solve this very technical problem microservices on the other hand they can help with technical things, but fundamentally they're not solving a technical problem. They're solving an organizational problem. They're solving the problem of too many cooks in the kitchen, too many engineers wanting to touch the same, uh, fundamentally the same software project. And what they allow you to do is decouple workflows and component life cycles in a way that increases the uh forgive the like agile speak for a moment but increases the velocity of your organization right so uh in

my opinion companies that are adopting microservice architectures are uh of a minimum size and i sort of point this out on the goKit website, there are probably at least 100 engineers. Maybe you can bring that down to 50, but probably not fewer than that. GoKit isn't really targeting organizations that are fewer than 50 engineers. Not to say you can't use it, but that's just not where we're kind of focusing our energy. And it's at that scale, in my opinion, that microservices really begin to make sense when you can decouple teams from each other's life cycles when you don't have to have these deployment dependencies when you can sort of treat other teams and other services within

your organization as uh in some sense a black box with an api that you can talk to and interact with to accomplish your own specific business goals. This is the context where GoKit is kind of designed to shine. So that's sort of microservice architecture. So let's pause there for just a second. So you mentioned microservices or service-oriented architecture. Some other options you have, monolith, SOA. Does it make sense to break down, I guess, the different options to developers and different options to those who are architecting services inside of organizations that are 50 to 100 or more developers?

Yeah, sure. Obviously, the buzzword is Monolith versus microservice, right? I just leaned out to a really awesome article. Not Chad Fowler martin fowler i think monolith first it was his uh the title of it so it's it's catchy yeah yeah and i think he's on to something um kind of glossing over the details in general i would agree that if you're a small team trying to find market fit or whatever you're doing with your venture capital money um it makes a lot of sense to start with a monolith because the velocity you can get in a,

but with four people crammed around a table is like super high. And microservices carry like real frictional costs that only make sense to pay when the benefits you get are bigger than the costs. And that only happens when your team is quite large, I think. Yeah, so the question is like sort of the evolution, right? I think when teams start on a product or an idea, monolith makes a lot of sense. And to be clear, monolith is the idea that all of your code is deployed in effectively a single binary as a single application to one or more application servers. And any change you make means you rebuild and redeploy the whole thing. Even if you just change the search function, you're going to rebuild and redeploy the entire application.

And that's fine for a long time, actually. I think a lot of companies can stay in that mode for years until they grow to a certain size. What ends up happening is when you get to a certain size, changes that you want to apply, and I feel like I'm just kind of reciting a history now that everybody has already heard a thousand times, but just for the sake of completeness, the changes that you want to make start conflicting with the changes that other people want to make. And you start having this sort of friction, deployment-related friction.

And at this point, you start thinking, wouldn't it be nice if I could deploy the search function totally independently of the user graph function or whatever the front end or whatever it may be. And when you start having these conversations around the water cooler, then microservices begin to make sense. There's a whole lot of costs that come with decoupling into independent processes that communicate over the network. The complexity is actually really, in my opinion, undervalued by people who are microservice champions.

But despite all that, I think they still do make sense once you get past a certain scale. And yeah, so that's the evolution as I see it. So that's that. And then you got the, I guess the focus here you said is 50, is $4.50. You've actually bent your own rules. You said 100 at first, and now you say 50 but that that's that's okay because we're trying to get go to a an even playing field and it's really focused towards the modern enterprise and you mentioned uh if those out there listening to this have heard your talk that you mentioned that it's not companies like apple ibm bank of america goldman sachs things like that it's it's not your large corporations

that typically you might think of when you think modern enterprise. It's more like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or even your previous alma mater, which is SoundCloud. And they sort of set the tone for the industry in terms of the way they build software that is consumer-focused and user experience-focused, And now you're growing your team. So when you had this idea for GoKit and it was back at Fosdum, you're thinking, well, what can I talk about? GoKit is what was born. What exactly is GoKit when you break it down? Yeah, so GoKit fundamentally is a collection of pieces that play well together,

but that you can kind of opt into one by one. And the idea is you are in an organization that's pretty big, and you're in a team that is deploying a bunch of services or microservices to accomplish specific business goals. So maybe you have a search service, maybe you have a user service, maybe you have an authentication service or something like this. These services, as they exist today, may be written in a number of languages, maybe Ruby, maybe Scala, whatever. And you have some idea that you'd like to use Go. And I think that idea makes a lot of sense because Go is, in my opinion and in many ways, sort of the perfect language for microservices. So what GoKit is,

is a collection of things you can kind of compose in and produce what I call a well-behaved microservice. And well-behaved means a lot of things. It means well-behaved sort of inside of itself. That means proper logging, proper telemetry and instrumentation, proper lifecycle management of components, stuff that keeps your process on your Linux system happy and keeps your telemetry and your metrics and sort of the business layer up to date and correct and that sort of thing. But playing nice also means playing nice with other services in your infrastructure. And so here there's a whole suite of things that are very subtle, very complex, and can fail in a lot of terrible ways.

And this is the field of distributed programming or distributed systems theory. And GoKit really simplifies a lot of it down. We choose as our messaging pattern, so-called messaging pattern, we're doing RPC right now. So if you know anything about distributed systems, you know that there's a lot of ways that processes can talk to other processes. PubSub, request response, async, message bus, blah, blah, blah. There's a whole litany of these like zero mq actually implements a lot of these and nano message the spiritual successor implements sort of a different set we've chosen to isolate down and focus purely on rpc for the time being emphasis on the time

being that's just sort of a optimization so that we can iterate quickly and get a product to market, so to speak, very quickly. So your service is going to play nicely on the system it's in. It's going to be an RPC service. It's going to call other services using the RPC messaging pattern, and it's going to play nicely with them. And so that means GoKit is going to give you idioms like circuit breakers on the client side, rate limiters on both client and server side. It's going to give you integration with service discovery components, whether you're using DNS records, DNS SRV records, console, etcd, or just manual manual configuration however you get services to find out about each other and talk to each other we're going to have integrations for that um we're

going to have integrations with uh distributed tracing systems in fact we already have an integration with zipkin which is the twitter sort of distributed tracing infrastructure we have several more sort of planned um and so it's things like this, things that are making services be well-behaved neighbors and these sort of infrastructures that I hope and I hope other people will agree is something that product owners and engineering directors want and need to see before they can kind of sign off on Go as an implementation language. This stuff exists in Scala in a number of different ways. For example, Twitter has something called Finagle, which is kind of what I'm driving towards. It's a similar collection of components, load balancers and so forth,

that make services play nicely in these kind of environments. Netflix has a whole suite of tools. I guess the most important one to search would be something called Ribbon, but there's all sorts of other correlated things that can compose together in the same way and kind of accomplish the same thing. Although I think most people believe that the Netflix stack is a little bit more tailor-made to the Netflix architecture. Twitter stack is a bit more generic.

And in that sense, GoKit is aiming to be a bit more generic even than finagle. There's a couple other ones that do this sort of thing piecemeal. Airbnb has something called Smart Stack, which is mostly concerned with service discovery and load balancing and that sort of thing. But that's a bit more infrastructural, a bit less libraries that you put in your service. So that's what GoKit is. It's a collection of these things that you can take in one by one, but that play nicely together and kind of give a positive story and a bright future to organizations that are buying into Go, perhaps for the first time or perhaps just more than they were before. Very interesting.

So I guess the question I think I come up with after this is, you mentioned a lot of big companies there with a lot of firepower. And I'm not saying you're one little guy, but I'm wondering what part Weavework plays into this. your ability to build what needs to be built to compete, even on a light scale to other organizations building similar things and other languages. Like, why will you win? Why will this, not so much win, but how will you succeed? Yeah, exactly. Being one person, how are you doing it?

Well, this really gets to the core of what I wanted to talk about today, actually, and what I'm really happy to be on the changelog for, which is the open source community, right? It's one of my favorite things about the Go community. And one of the things that probably has kept me around for longer than I've been part of anything in sort of the technology sphere is that the quality and the intelligence and the friendliness and just the level of people involved in the Go open source community has been one of the nicest things for me and nothing I've seen in any other sort of sphere that I've been in. And so you asked me, I'm just one guy. That's definitely true. This is absolutely a passion project. It's something that I want to see

happen. It's not something that I'm being funded for. WeWorks is not directly sponsoring any of this work. It's really a nights and weekends sort of gig for me. And I'm motivated purely out of sort of this, hold on, I want to make sure I say the right word, benevolence. That's the good one, right? Malevolence is the bad one. Yeah. Okay. Oh, is it? I didn't know there was a bad or good.

Educate me. Well, whichever one is good, that's the one that I have in my heart. Yeah. And so in combination, so I feel like my job is kind of to carry the banner and say, hey, if Go is going to get to the next level of its success, which I'm sure it will. And I and I hope it will. We're going to need stories like GoKit. We're going to need things like GoKit to get it there. We're going to need buy in in sort of like the next tier of developers, which are the developers that are working at these somewhat larger companies that maybe aren't refreshing hacker news eight times a day and that still like

deserve the benefits that go can provide so i'm kind of viewing myself as um at the moment definitely the primary developer but more of a sort of community leader and and i'm hoping uh with the help of the change log with the help of the talk that I'm going to do at gopher con in about a week. And, um, with the sort of publicity for lack of a better word than I'm going to be doing throughout the year to, uh, attract people who have similar ideas. And I definitely got some of this, uh, when I gave the two talks in Brussels and in London, I was approached by several people in those communities and saying, you know, I've been saying the same thing to my guys at my organization, my girls at my university. And we really feel that the time is right for this and we're excited to contribute. And

indeed, we've had a couple of really great, powerful contributors so far. Top of my mind is Chris Hines, who's been really incredible in helping me with the log package. We've iterated on a few API designs. We're continuing to iterate. I think we're going to produce definitely, even if you use nothing else in GoKit, the GoKit log package is going to be

the premier value-add log package in the Go universe. And there's already, that's a very crowded field. So it's a big statement, but I think it's going to get there, absolutely. My friend here in Berlin, Tomas Sinart, has helped me a lot with iterating on the endpoint API design

and like the core kind of components there. Canonical friend of mine, Roger Pepe, I hope I'm saying his name correctly. He's helped me a lot with the rate limiter. And we have a core stable of contributors. A couple of people from DigitalOcean are very interested as well and have helped me out. So I hope this grows.

And you're absolutely right to say just one guy isn't going to be able to produce code to the same level that Twitter is able to do. But I hope that my instinct is correct and that this is something that people want. And with a little bit of help, I'll be able to attract the kind of people that can make patches and push it forward with me. Absolutely. Well, that's what we're here for. And I'm glad you mentioned DigitalOcean because it is time for a sponsor break. And I'm going to change the order of it.

TopTile was supposed to be the first sponsor of the show, but, well, technically the slot number two. But I'm going to move DigitalOcean up just because you mentioned them. So we're going to take a break and hear from our friends at DigitalOcean who love this show and obviously love what Peter's doing with GoKit and is supporting it. So take a listen. We'll be right back. I have yet to meet a single person who doesn't love DigitalOcean. If you've tried DigitalOcean,

you know how awesome it is. And here at the Changelog, everything we have runs on blazing fast SSD cloud servers from DigitalOcean. And I want you to use the code Changelog when you sign up today to get a free month. Run a server with 1 gig of RAM and 30 gigs of SSD drive space totally for free on DigitalOcean.

Use the code CHANGELOG. Again, that code is CHANGELOG. Use that when you sign up for a new account. Head to DigitalOcean.com to sign up and tell them that Changelog sent you. All right, we're back. And I guess, Peter, since you mentioned DigitalOcean, we just had that super awesome sponsor spot for DigitalOcean. And you also mentioned your upcoming talk, which is we're recording this.

Let's see, what's today? Today is June 30th. Yeah. And we'll publish this on July 3rd. And on July 8th or 9th, I'm not sure which day you speak, you'll be at GopherCon. You'll be talking about GoKits. So on one of those days, we'll be giving a talk going even more in depth. So if you're a Changelove listener and you're not going to GopherCon, there's still probably about a half a second to buy a ticket. Maybe they're all gone.

I don't know. But we'd love to see you there. And if you see us, say hello. But we work with GopherCon. We're doing video work with them this year. If you see us running around, we've got cameras in our hands. Say hello. It's me, myself, which is me and myself, right?

We've got Jared, and we also have Donald. I call him DK. But if you see us around, say hello. Well, every day we have plenty of them, but every day we'll be wearing a ChangeLog t-shirt because what other uniform will we wear? But you'll be there. GopherCon is awesome. 1,500 people. How excited are you to come back over to the U.S.?

You were just, what, in California recently, right? Yeah, that's right. I was just there for DockerCon. And I think that was, I don't know how many people that was, but it was on the same order of magnitude. DockerCon was huge. It was really a gigantic, gigantic conference. I'm looking forward to, I've never been to, well, I guess I have. I've been to RailsConf and i have i've been a rails comp and stuff like that and i think that might have borderlined on like 800 900 ish at the time now it's much more um but it's been a while since i've been to such a large conference and then to

be in the role of documenting what's happening it kind of blows my mind that we'll have to like catch up with 1500 people somehow yeah it's gonna be some way but but anyway it's enough to say gopher con's awesome you'll be there you'll be speaking there that's right and that's uh yet another plug but let's dive deep uh now that we're back from that break let's dive deep into go kit and its components and their statuses and and kind of what what up GoKit itself, not just this idea of it. Where should we begin there? So probably conceptually the place that makes sense to start is package endpoint,

which is kind of, there's not much there. I think there's like three things there. But the idea is to define a common interface or a common function signature that everything else can be built around. And this gets back to what I said. The way we're starting is by assuming RPC, the RPC messaging pattern. So in package endpoint, we have what I was able to figure out

as the lowest common denominator RPC method signature. And it's kind of what you'd expect. Funk takes a request, returns a response. But because it's in Go, it returns a response and an error. And because we need to thread a lot of things through this, it also takes a context. And a context is a thing that was released not too long ago, maybe a year, maybe a little bit more, from Google. It's this value add package, and it is a little bit awkward when you first see it. But it's a parameter that you can thread through, or rather you should thread through, all of the functions or all of the points in your request path. And it allows you to do a lot of really handy stuff.

At a basic level, it allows you to thread information sort of across stack boundaries, across thread boundaries, basically across bounded contexts within not only a single process, but from one service to another. And it also lets you set up nice, I forget exactly what the term is, sort of like a hierarchy of component ownership, process ownership, lifecycle ownership. So if you're a component and you want to scatter-gather 10 requests

to 10 other components, you sort of can set a sort of timer, send those requests off. And then the first one that comes back can, for example, cancel all the other ones, and they'll terminate and clean up nicely. So the context is something that gives these sort of nice semantics. And it's been around out in the wild for, I guess, about a year. A lot of things have made good use of it. And so GoKit has chosen that as its mechanism of taking information across context boundaries. So that's a good place to start. And once you get your head around that, everything else kind of opens up like a flower.

So you've got package endpoint in no particular order, but I'm going to read them down from the readme. Package log, package metrics, package endpoint, which we just talked about. I'll not say package anymore because it prefaces all of them, but transport, circuit breaker, which we sort of talked about a little bit in theory earlier, load balancer, rate limiting, or rate limit. Those are all implemented. And then in prototyping, you've got tracing. You've also got client patterns, which seems to be, I'm not really sure what that makes up,

but you've got service discovery, which is impending. And then you've got some other, I'm sure, some other ideas. Ad services is implemented as well. Yep. So let's start at the top, I guess. Log and metrics are maybe the simplest ones in the sense that they just do a very specific thing. Package log is like many other log packages kind on and that was in the initial RFC, which was contributed by a DigitalOcean guy, by the way, was that microservices or rather package log in Go kit enforces strictly the idea of structured logging. So if you're familiar with the standard Go logging package, you can write log.printf and then just sort of an arbitrary string. And it's GoKit's opinion that that's kind of bad practice. And what all of your logs should look like is key value pairs. So if you wanted to log, for example, starting equals blah, blah, blah, and then

log that sort of collection of pairs. And you do pay a cost in sort of human readability, but it's Gokit's opinion that that cost is more than made up for in the ability to programmatically read those logs, parse those logs, and hopefully at 2am, God forbid, make sense of those logs. So that's sort of the one core opinion about log. And then we have a lot of stuff that kind of wraps that core opinion and gives nice value add stuff like leveled logging is in the pipeline, contextual logging, different output formats. Another important thing about package log is that it's not only for application logging, it's also equally usable for so-called structured log format data. This is the kind of stuff that you might push into a Kafka instance. This is click

tracking. This is general analytics. It's anything that needs to have a stronger sort of QoS than simple application logging that might end up in an Elk stack, an Elasticsearch log stash Kibana stack. So log is good for all of these things. A natural sister or brother would be metrics. Precisely. And metrics is also simple in the sense that it does sort of one thing that's independent from other things. It lives in your process. And actually, maybe we should take a step back and ask, like, what is metrics?

What is instrumentation? Lots of people have. I love your question. Do it. Yeah, yeah. There are different ideas. There are services like Airbrake, right, where you sort of trap all the exceptions or errors in your code. And then whenever you see one, you emit a piece of information to a third party server that collects them and tells you what parts of your system are crashing or whatever.

That's kind of a type of instrumentation, I guess, but it's not what metrics is. Something a bit closer would be a system like Graphite or StatsD, where rather than trapping errors, what you're doing is going through your code and you're instrumenting all of the important bits. And what is an important bit? One thing might be the number of requests that hit you on an HTTP service, the duration of those requests. So average, mean, median, max, min, this sort of thing. Not only the basic sort of statistics, but also bucketed over quantiles. So mean 50th percentile latency, 99th percentile latency, this sort of thing. So that's kind of more what I'm getting at here. And metrics, package metrics in GoKit represent sort of a distilled, boiled down version

of what I and many of my contributors have found is important. We expose three core concepts, that is the counter, the histogram, and the gauge. And we provide different core concepts that is the counter the histogram and the gauge and we provide different back ends that implement each of these counters you can or each of these metrics you can hook them all up together in a single sort of api you can use one or as many as you want and so the idea is you should aggressively instrument your code using the package metric sort of interfaces. And then once that program startup in your funk main, you wire up the interface to the back end of your choice. And this is in keeping with our sort of philosophy, go kit philosophy of working with the infrastructure that you have. You almost certainly as an organization are going to have a StatsD server or a Graphite server or whatever it is

existing and receiving metrics if you're a team of 50 engineers. So we want to work with that. And we have adapters for many of the common ones. The one I'll plug is Prometheus that was also developed at SoundCloud. And yeah, we think that's probably the best model for this kind of thing yeah we've heard good things about prometheus as well around here we've been meaning to to to do a bit more coverage on it too so definitely if you need an intro let me know i know the developers quite well and they're also going to be at gophercom well that would be awesome let's make it happen yeah definitely

no it's a really cool piece of software and it does its job quite well so adapters into some of the most common metric packages is the exp var is how you pronounce it yeah well xvar is one of the standard library packages in go and it's really like bare bones simple stuff but pretty cool you can export kind of instantaneous view of certain certain types of metrics and just kind of dump them by hitting a specific HTTP endpoint. So it's a good entry-level bare-bones type of exposition. In development or something like that. Yeah, sure, sure. Gotcha.

Okay. Very cool. That makes sense. So counters, gauges, histograms, those all dump data into these known metrics packages. Precisely. That help you sort of dive deeper into them as you need to. What's next? So if we walk down the stack, we can kind of start looking at the value add components.

And this is stuff that benefits from the common endpoint API or the common endpoint interface. This is things like circuit breaker, load balancer, rate limiting. And these are things that you probably can intuitively guess that a microservice needs, but it actually turns out it takes some thinking to get it right. And this is part of the value proposition of GoKit. We have contributors who have thought about it and have made mistakes implementing circuit breakers and load balancers and prevent you from making the same mistakes, hopefully. So each of them is something that you want to wire into your microservice, either at the client side when you're connecting to other services or on the server side when you're receiving connections from other services. And they prevent you from behaving badly. So let's start with Circuit Breaker.

This is something that you would typically put in the client side. And what it does is if it detects that requests to a specific backend, to a specific other service are failing regularly above some threshold, let's say, it will prevent other requests from going out until it detects sort of a healthy state. And the idea here, the reason it's called circuit breaker is it's kind of like a fuse, right? If you put too many amps through a fuse, it's going to explode and prevent you from starting a fire in your house, right? Great naming. I like the naming. Yeah, yeah. It's a cool name. It's a pretty common pattern.

But until you've had this sort of thundering herd where a failure in one service brings down your entire company and it takes like hours to get everything restarted and back online, you don't really understand how important it is to have these things. And it is important. And GoKit ships with one. Actually, several. You can kind of pick your favorite type of implementation and wire it into hopefully every request that you make.

And it's important to note that circuit breakers aren't really a mechanism of solving load problems. So if you're under-provisioned in your infrastructure, circuit breakers aren't going to fix that. What they do is prevent bad problems from becoming terrible problems. And for that reason, they're very important. Does this tie into load balancer then? Yeah, in a way, you would certainly wire circuit breakers and load balancers together, where circuit breaker is kind of something that only comes into play when things go wrong. Load balancer is something that's in play all the time when things are going right. The idea is that if you have a set of services that are horizontally scaled, multiple instances of the same thing to support the amount, the millions of requests per second that I'm sure

your startup is getting, load balancer is something you're going to install in your client side to distribute the load across all of those instances. And there's a lot of ways you can do that. You can do very simple things like picking a random instance every time. You can do well-known algorithms like RoundRobin. You can use an algorithm that sort of weights each instance based on criteria like average response times. You send more requests to the healthier instances. You can weight them based on locality, based on data center. So Packaged Load Balancer is something that allows you to implement those types of algorithms across sets of otherwise identical instances of services.

And we provide a lot of hooks for getting sets of instances. And so this is sort of the service discovery component. Given you know you want to talk to the user service, and how do you translate the string user service to a set of instances? Well, this is a whole, probably a multi hour changelog podcast in itself. But we provide hooks for a number of common solutions to that problem. And so that's all sort of wrapped up into the load balancer package. How is that different from rate limiting? So rate limiting is something that you can stick either on the client side or the server side of your service. And it's something that maybe is a little bit less useful

or at least less generally applicable. But if you know that, for example, you're calling out to the Facebook API and you know that you don't want to do more than 10 requests a minute or 100 requests a second or whatever it happens to be, you can declare that upfront. You can stick a rate limiter on that client connection

and you can say, I only want to do 100 requests a second max. And then you can determine what to do with any requests that happen to go above that limit. You can kill them immediately with an error. You can tell them to wait a while, whatever is right for you. That package is pretty bare bones at the moment, but there's hooks for doing things like hooking into a central lock store so that you can enforce, for example, a consistent rate limit across multiple instances. Yeah, the sky's the limit there, but that's what that does. So just to summarize here, we got log metrics, endpoint, transport, circuit breaker, load balancer, and rate limit. And those are all implemented and ready to use today. And tracing is in prototyping stage along with client patterns. Can we talk

through those two pieces there? Sure. So tracing is something that maybe a lot of people, some people know about, some people don't. The idea is when you have a microservice architecture and you have a bunch of things talking to each other, typically what happens is a request is going to hit your website, for example. It's going to go to some service. That service is going to need information from a bunch of other services. Each of them in turn may need information from more services. And so what happens is you create this sort of tree, this sort of hierarchy of requests that was all spawned by a single incoming request. And once you get past a certain size or a certain level of complexity, it's really important to be able to look at that call graph and see how long each individual thing takes,

where your hotspots are, this sort of thing. And the way to do that is with a so-called distributed tracing framework. The canonical one in the open source world is, well, I should take a step back. This all became sort of public knowledge quite a long time ago, I think maybe 10 years ago at this point, when Google released a paper called Dapper. Dapper is the name of the Google internal system that does this. And not to say it was the first system that does this, but that's sort of when it became en vogue. And so in Google fashion, they released the paper, but not the implementation. The Apache folks, I think, maybe it was Twitter initially. I don't have my history right on this. Somebody said about creating an open

source implementation of the Dapper paper, which when complete, they called Zipkin. And Zipkin is what you can kind of download and use today if you're running on the JVM. There's a number, I guess that was three or four years ago that that became public knowledge. There's a number of similar systems now in other languages. There's a number of Zipkin implementations in other languages. And so GoKit provides a package tracing that does this sort of thing. We have a Zipkin implementation at the moment. So if you have a Zipkin infrastructure

in your organization, we can interact with that. We have planned support for a similar system from a company called Sourcegraph, which is also another prominent member of the Go community. They have a system called AppDash, which does basically the same thing as far as I understand. I plan on tackling that at some point in the near future. And there's actually a couple of other ones. There's another, I think, Apache project called hTrace. There's something from Netflix

with a funny name that begins with S that I can't remember. Actually, during GopherCon, there's a distributed tracing working group that's going on in Budapest that unfortunately I'm not going to be able to be a part of. But they're sort of having a symposium and talking about the future of this sort of thing. So keep an eye out in the next month or two. I'm sure we'll see some interesting news there. Very interesting. So a lot of stuff happening around that. I mean, especially the support with a lot of new things I've heard there too. Zipkin, AppDash, Dapper, and like you mentioned, it is

similar to how Kubernetes came out recently. It came out as a paper first and well, sort of, what was the, Kubernetes is its own thing, but then the other thing they recently announced. Borg, yeah, Borg. Yeah. That's what I was thinking. Thank you for helping me there. It's like they release this paper,

but then you also have Kubernetes, which is similar to what's happened with Borg. Exactly. I think they would say that Kubernetes is like the, if they were able to redo Borg and fix all of the problems that they discovered, that that would be where Kubernetes is like the, if they were able to redo Borg and fix all of the problems that they discovered, that that would be where Kubernetes is. And kind of made a bit more straightforward for the open source kind of public world at large. Yeah, exactly.

You got Zipkin, AppDash, and what was the other one? HTrace. And then there was another one, the thing you mentioned, but I didn't get a chance to jot that one down. Yeah, I didn't get a chance to remember the name of it either. All right, we'll go back and list it. We'll get it in the show notes. I was just thinking for the show notes' sakes to make sure we get it all in there. Yeah. Let's see.

Where are we at on the list here? I guess I closed that. I closed that list. Let me get back to the readme so I can get back there. Next is, is client patterns. Exactly. This has no link. There's no description.

Help me out here. So, yeah, the idea is you write a service, and your service has an implementation, and people can call it. And this is something I haven't mentioned yet, actually, but it's actually also pretty core to the GoKit idea. You're going to have a service and it's going to be implemented in Go. And you're going to do your implementation once. But you probably want to be able to expose that service on any number of different transports. Maybe your company is using Thrift exclusively behind the scenes. Maybe it's using just standard HTTP, JSON semantics. Maybe it's using a bleeding edge, uh, G RPC, which is actually very similar to Kubernetes. It's like the open source, the Google open source version of an internal project called stubby, uh, an internal

RPC framework. So I don't know, we, we have a bunch of, uh of so-called transports that allow you to expose your service in different ways. And a core idea of GoKit is that you should write your implementation once using sort of the RPC pattern, but then you should be able to expose it on an arbitrary number of transports simultaneously in the same process. And so, yeah, that's an important thing that I sort of forgot to mention. So in support of that, you have the service running on some transport and you can hand write the code to talk to that. If it's HTTP JSON, it's probably pretty straightforward.

But if it's a thrift server, for example, it's a little bit more laborious. And so client patterns is just my way of saying, given you're exposing a service on any one of our supported transports, we want to be able to show you an example client you can build that gives you the same semantics, the same Go language level RPC semantics to talk to that service over the chosen transport. So if you have some service that adds two numbers together and you expose it on Thrift, gRPC, and HTTP, then we want to be able to say you can create a Thrift, gRPC, or HTTP client that can talk to that service. And you just get the really straightforward, basic call response semantics on it. So client patterns is just building those things up, basically, and making good examples.

It sounds like service discovery might play a little bit into that in terms of discovering different services. Exactly, exactly. So yeah, it's a bit of a tautology. But yeah, precisely, you're going to, as part of a client package, you're going to figure out how to get to the services you want to talk to. And that's a question that's answered differently in different infrastructures. But yeah, absolutely.

You're going to wire in service discovery into your load balancer all on the client side and then use that in combination with Circuit Breaker or whatever else to talk to those remote services. That's interesting. I mean, as we walk down this entire component status list and everything that you're doing with it, I mean, it starts to really make a lot more sense in terms of what GoKit is trying to accomplish and reminding all the listeners, too, that quite a bit of this is implemented we didn't even talk about the api stability document and that that piece yet there but uh if um if you want to talk about ad service real quick and then just sort of go through the api stability which also talking about these statuses you got adopted implemented prototyping pending uh yeah so

there's several different statuses that it's sort of tricky to to navigate around as a as an outsider yeah yeah um and that's totally my bad this is all sort of in a pre-alpha state and the words don't really mean a lot well you are one person we're trying to we're giving you some slack here so we're not holding your feet to the fire peter. Yeah, much appreciated. Yeah, it's just a sort of a signal to anyone who might stumble by as to where we are sort of in a percentage-wise in accomplishing the goals we set out for ourselves. So yeah, pretty much everything is implemented with a basic API. We're not personally GoKit.

Like GoKit itself is not in an API stable state. If someone came along, filed an issue and said, hey, here's a much better API for the metrics package, take a look. If indeed they're right, then that's going to change. And yeah, we're going to be that way for a little while. So this is definitely like alpha quality stuff right now, alpha stage stuff. But that said, the example ad service is sort of the proving ground for a lot of this. It's just a directory of a fake microservice that implements all of the transports. It implements all of the little value add composed in features just as a way to feel, get a sense of how all these things play together.

And, um, it's, it's really an opportunity for me to see if I'm accomplishing my goals with this thing. If, if I design my APIs correctly, this stuff should sort of fall out naturally and feel very good to, to write on the page. And if that's not the case, I'm going to see it in ad service and I'm going to need to change things. That's interesting. So the ad service essentially is a good example for anyone following in our footsteps of this conversation to look at how it should be implemented or how it should work. Exactly. And as you're working sort of test privy rounds. Yeah, exactly. And I think the thing I'm most proud of that is that when you look at the the main function in ad service there's no magic at all it's very uh comparatively long but it's all straightforward nothing

nothing like there's no package globals that get sort of magically pulled in everything is composed in this very declarative bump bump bump step by step way that is really, for me, a joy to read. It's so easy to figure out what's being wired together, how the information is flowing through the object graph. This is really nice for me and something I'm really seeking to preserve. On the notion of the API stability, you'd mentioned that the API is stable, but it's not, like, in quotes, stable to the point where you couldn't come by and change it if someone wanted to listen to this podcast or

pick up on what you're doing and step in and help out in some way. Does it make sense to talk about that API stable quote you have on the docs there that you wrote there? Do you know it by heart? Let me read it for you. No, no, I've got it in front of me too. Um, and this is actually sort of a conversation that touches in the broader go ecosystem. Um, one of the most longstanding sort of trouble spots or maybe concerns in the go world is,

uh, this idea of how do you manage dependencies and how do you make reproducible builds and how do you enforce things like API stability policies? And there's been a number of solutions to that that have been more or less successful. Luckily, in Go 1.5, which is due for release, I think in about a month, we have sort of a canonized, blessed approach to this process of so-called vendoring that's going to be baked into the Go tool. But for a Go kit, really, the API stability policy comes in two parts. It says, first of all, if we're going to bring in a dependency, we'd prefer to have one that has a stable API so that users of our software don't get unexpected

breakage whenever they import us and we in turn import something else. So we prefer to do that. Given a choice between two packages, we'll pick the one with the stable API. The other side of the coin is the API of GoKit itself is not currently stable. So if you want to use GoKit, what you should do is vendor it in. And this is sort of a process by which in your service, you shouldn't import GitHub GoKit kit sort of directly. You should use some sort of vendoring tool or some sort of vendoring process to vendor that code into your repo directly and use it from that place so that you control the lifecycle.

I guess let's take a pause there. We'll have one last sponsor break. When we come back, we'll talk a bit, I guess, about the closing pieces of it. I want to talk to you about the potential working group and what's going on at GopherCon to see if there's any sort of get together for go kid enthusiasts or those who want to step up and help out so let's break we'll come back we'll talk about that top towel is by far the best place to work as a freelance software developer i had a chance to sit down on top of brendan beneshot the co-founder and coo of top towel and i asked brendan to share some details about the foundation of TopTile.

What makes TopTile different and what makes their network of elite engineers so strong? Take a listen. I mean I'm one of the co-founders and I'm an engineer. I studied chemical engineering and to pay for this super expensive degree I was freelancing as a software developer and by the time I finished realized that being a software developer. And by the time I finished, realized that being a software developer was pretty awesome. And so I kept doing that. And my co-founder is in a similar situation as well. And so we wanted to solve a problem as engineers and do it as a network of engineers,

kind of for engineers by engineers. And having that perspective and consistently bringing on new team members who also share this really makes TopTel different in that it's a network of engineers, not kind of like you have TopTel and then the developers. It's never about us and them. It's always us. Like everybody at TopTel, for the most part, refers to TopTel as their company and they feel feel like it's their company, and everybody acts like a core team member, even though they're freelancers within the TopTal network. And all of these things are extremely important to us.

All right, if you're interested in learning more about what TopTal is all about, head to toptal.com slash developers. That's T-O-P-T-A-L dot com slash developers to learn more, and make sure you tell them the change law sent you all right we're back uh i guess peter this has been quite an enlightening trip down go kit slash go slash microservices lane good to hear um i know i've certainly learned quite a bit and i'm thinking for the listeners sake you know how can they get involved i know that's one of our coin questions at the tail end of this podcast which we'll get to here in just a

minute or so but um i'm thinking is there a working group at fosdum you got a lot of enthusiasm rongo kid and i'm wondering what's happened since then that was february this is obviously june now so not too much time but enough to have some things change and get more. You've got Chris and you've got some others. So who else is sort of stepping up and what's happening in terms of a working group? Yeah, so there was a lot of ideas that came into the sphere right away. We set up a mailing list, and so that would be a good place to start you can see that sort of i think at the top of the readme or at the top of the website uh you can subscribe

to that it's just a typical uh google group mailing list and um so that's that's good for a sort of longer form discussion there's also a slack channel uh which we set up recently on the gophers.slack.com slack group. Is that how it's called? Slack organization? I have no idea what to call them these days. They're just out there. There's so many slack rooms, man. There's so many things. So many.

But yeah, that's actually a well-managed and moderated slack group. And there's a channel in there, GoKit. You have to get an invite for that but invites are freely available and i provide a link for invites on the go kit website as well what's the URL for that right now real quick so i can write it down gophers.slack.com okay and if you want an invite that is bit.ly slash go-slack-signup. And that'll get you one in an email right away, as far as I'm aware. Awesome. Yeah, so those are the online forums.

I'm pretty active on the Slack thing, so if you have ideas, that's probably the easiest way to reach me. And there's also several contributors that hang out there. In terms of GopherCon, we're definitely going to do a GoKit sort of hack day on the hack day. I think that's going to be Wednesday. Is that right?

Thursday. Tuesday is the training. I think they're calling them workshops. So that's Tuesday. Wednesday, Thursday is the conference days, and Friday is hack day. Yeah, so I was at the hack day last year, and that was actually the the probably the

coolest part of the conference uh because it was an opportunity to meet people you kind of only knew by their twitter or github handles or whatever and uh the level of stuff that was produced there was actually really cool i'm not really much of a hackathon guy myself but uh the hack day was a much different experience than that it was a bit more heads down, a bit more focused, and really like a good sort of atmosphere and a really productive space. So I'm really looking forward to that. We're going to have a go-kart table. We had Eric and Brian on the show recently, and on that note, they said that, you know, you mentioned Hackathon, that it's not hackathon at all. It's more like for those 1,500 people come to the conference, you got all these speakers and people there that are building

and contributing into the Go ecosystem. This is a chance to sit down with, you know, your, like you had mentioned, the avatar you only know, or your hero, so to speak, and sit down and hack with them on your favorite library or ask them questions and sort of fiddle with code together. And that's more or less what you got out of as well? Yeah, definitely.

I met some really cool people there for the first time who I'm still talking to on a weekly basis today. And I'm looking forward to meeting them again this year. Very cool. So yeah, Chris Hines mentioned in the mailing list room or your list, that a GoKit Birds of a Feather might happen. So is that around the same hack day kind of thing? Yeah, as far as I understand. I don't know if there's a formal process.

If there is, I'll look into it and I'll be sure to set something up. But I think it's pretty informal. Maybe day of, I'll set something up and I'll make myself visible. So yeah, if anybody's at all interested, show up in the right room and you'll be able to find me. I'll make sure that's possible. And while we're talking about GopherCon, if you're listening to this and you're not at GopherCon, but you know somebody who is, and they listen to this show and they want to get on camera, they want to say hello, we'll be at the first after party, the second after party, the hack day, pretty much everything you see there will be there. Cameras in hand, The Change Log has a new division called Change Log Films.

We're going to conferences and helping conferences like GopherCon document what's going on in the community. And it's a huge part of what we're moving towards. So if you're listening to this and you're not there, hopefully we'll see you next year. But if you've got friends that are there, hit them up. Let them know we're there. If they don't already know, Tell them to come check us out because we want to see everybody. We'll have you on camera as well at some point, Peter.

You can't hide. You can't hide. I guess in true fashion at the ending of this show, let's wrap with two awesome questions. The first question is if we haven't already said it, obviously, we've talked quite a bit about several things instead of Go and Go Kit.

But for those out there that are thinking, man, this is super cool. How can I get involved? Where are the places you need help to make Go Kit a real thing and keep moving forward and get better adoption? Yeah. So this is a great question. If you're an accomplished Go developer and everything I've been saying has really struck a chord with you, then dive right into the issues list. I've been making an effort and I'll continue to make an effort to keep my roadmap up to date in there and have a list of

things that need to be implemented that I'll try to make as straightforward as possible. If you want to claim ownership of them, please feel free. A lot of them is pretty straightforward. A lot of them are going to be a little bit subtle, but it's nothing we can't talk through. So if you're a gopher already and you want to contribute, that would be amazing. But that's not even really necessary. The thing that I think would help quite a lot, and especially at this early stage, is if you're somebody in an organization and you want to use Go, but you're feeling some friction or you're missing something in the ecosystem, something that GoKit might be able to provide or even help provide, file an issue with that. Let me know somehow. And let me sort of fold it into my idea of what this thing is that I'm building. Use cases like that. User stories. Sorry again for the Agile terminology. I've.com too so you said you're there all the time and so i guess issues would be a good place to go and kind of uh find things off yourself but at the same time get an invite yep precisely in that slack room precisely awesome well cool and

i guess our our final question which to some i don't know peter you might like it as well but some absolutely hate this question some love it we'll see what line you fall upon but uh we're curious who your programming hero or heroes are yeah that's interesting it's sort of a like with me it's it's it's like a favorites like i try not to have favorites i try not to do things like that, but I will say that there's been several people in my career that have been very heroic to me. And it's not specific people or specific personalities. Rather, it's been people who have been mentors to me, not only when I was in school or fresh out of school, although those people have played like a really important role in growing me and making me a better person and a better programmer and all these fun things. But even like on a day-to-day basis today, people who take time out of their day to contribute to me and

to like lend a little wisdom to me, if you can get into a mentor, mentee, is that right? Apprentice, master sort of thing, this kind of relationship with somebody in a professional context, I think this is really the way that information flows. This is really the way people get better. And to everyone who's ever been a mentor to me, that's people like Gary Sumar back at Bloomberg, Sean Treadway at SoundCloud, more people than I could possibly name, actually. But these are the people that are like heroes to me and the people that have really allowed me to level up and get to where I am today. Speaking of mentoring and menteeing, I'm not sure that's the thing or not. Maybe not.

I'm going to follow your lead on that one. Are you mentoring anybody? Not sort of officially right now. And that's a great sort of call out of me. I should really thing or not maybe not i'm gonna follow your lead on that one are you mentoring anybody uh not sort of officially right now and that's a great sort of call out of me i should really find a way to do that um yeah i'm gonna i'm gonna look into the community here because berlin is a great place for this lots of junior devs lots of people looking to get into the industry so yeah it's definitely something i could look into yeah uh i'm trying to remember the the ad spot i did for digital ocean but uh in that ad spot i talked about the huge community for um startups there because they just opened up uh fra1 yeah frankfurt yeah so yeah and it's on that trying to remember the words for but some some exchange

some internet exchange that's huge that feeds basically all of europe and it makes it super fast that's where the digitalOcean servers are at. Not to plug them much more, but you made me think about how that spot talked about the thriving startup community there and ecosystem there. Yeah, definitely. And when they came and did their press tour, I was actually helping them coordinate that. There's a lot of fun stuff happening in Germany and Berlin especially. So yeah, definitely.

And you said there's some folks at DigitalOcean helping with GoKit? Yeah, there's a couple of infrastructure engineers who have contributed to the log package a bit to the server side stuff. With luck, they've already contributed a sort of a staging testing server for me to help me test the Zipkin stuff. It turns out if you want to start a Zipkin server, you need a pretty beefy machine. Zipkin is written on the JVM, and it uses a lot of memory, more than my little dinky VPSs can handle.

So thanks to them for that. With luck, that'll continue. Yeah, so DigitalOcean has been a great partner for GoKit so far. I guess the one question that eluded my mind to even think about but mentioning digital ocean there and asking about their uh work into the into the software itself is who is there anybody out there who's adopted go kit so far and using go kit in the

wild or even in like a test phase yeah um there's a couple of organizations that I can't really name by name for a variety of reasons that are using pieces of GoKit. The log package is attracting a lot of attention. Metrics is attracting a lot of attention because these are sort of the most feature-complete, kind of self-contained packages at the moment. But yeah, a couple of others are starting to play around with structuring microservices using GoKit components. And then hopefully the goal is their feedback is going to drive further GoKit development. So that's the idea. Fantastic.

Well, Peter, it's been such an honor to have you on the show, man. I know we've been playing Twitter DM tag for a little bit and then email a little bit and then you were traveling and then it was a good time for us. So we finally got you on and we wanted to get you on the show after we had that conversation with Andrew. Yeah. Because that sort of set the new tone for Go on the show.

Go in the large. Yeah, exactly. We sort of stepped back into Go on a year-to-year basis. And we started with Rob Pike by himself way back when go very first started and then about two years back had andrew and rob back on and then we had andrew back on and i knew that it was time to to to talk to you because he was like hey when i'm done on this show you got to get peter on the line because he's got something very cool happening that

everyone in the go community has to know about nice so so there you go so you got a good old uh slap on the back and blessing from andrew nice that one as well good to hear and i and i hope to hear even more go people uh on the change log in the future i'm definitely going to tune in if that happens we definitely want to talk about prometheus at some point so when we get off here we'll have to get some get something happen oh yeah definitely in the near future but peter it's it's been awesome any links you want to mention as we close here to follow you on then you got your github.com slash peter and your last name anywhere else it's best to kind of catch up with you at twitter yeah i mean i'm probably easiest to reach on twitter that's just my full name peter Bragon. No spaces.

GitHub, sorry, gokit.io takes you to the GitHub repo. Yeah, I guess that's about it. I'm a pretty simple guy. And for the listeners' sake, we'll have all those in the show notes. So this is episode 163 of the ChangeLog.

Go to changelog.com slash 163 and you'll find all the show notes and all the details about everything we talked here so don't feel like you got to wreck your car if you're listening in the car or jumping out of that airplane to get to i don't know just making funny jokes but don't go crazy just go to the show notes everything's there for you we make it easy but uh peter thanks so much for joining us today on the show and uh let's let's say goodbye all right yeah no thank you very much the pleasure is all mine and i had a lot of fun you We'll see you next time. you