The Changelog: Software Development, Open Source - Jenkins and Continous Integration (Interview)

Episode Date: February 8, 2011

Kenneth and Wynn caught up with Kohsuke Kawaguchi and Andrew Bayer from the Jenkins project to talk about continuous integration, Java, and corporate backing drama.

Welcome to the Changelog episode 0.4.8. I'm Adam Stachowiak. And I'm Winn Netherland. This is the Changelog. We cover what's fresh and new in open source. If you found us on iTunes, we're also on the web at thechangelog.com. We're also up on GitHub. Head to github.com slash explore. You'll find some training repos, some feature repos from the blog, as well as our audio podcast. And if you're on Twitter, follow Change Log Show.

And me, Adam Stack. And I'm Penguin, P-E-N-G-W-Y-N-N. This episode is sponsored by GitHub Jobs. Head to thechangelog.com slash jobs to get started. If you'd like us to feature your job on the show, select advertise on the changelog, and we'll take care of the rest. Folks at PandaStream need a special dev

who's at ease with Ruby, Redis, MongoDB, and Beanstalk in production, preferably in the U.S. as a remote worker, but also in London, if you happen to be there. If you're interested, lg.gd slash 6z. And if you're an Objective-C, Cocoa, or iOS dev that likes working with really smart people, check out Mutual Mobile

based in Austin, Texas. They want to talk to you. Check out lg.gd slash 82. And Store Envy, aka the Tumblr for e-commerce, is looking for a senior Rails dev with JavaScript, Redis, NCacheD, and MySQL Chops. If you're interested, NCacheD, and MySQL chops. If you're interested,

lg.gd slash 6l. StoryMV made those awesome changelog tees we handed out at South By last year. Those are awesome tees, man. Now we're eagerly awaiting our stickers from Sticker Mule. Die-cut stickers.

Can't wait to put one of those on my Mac. I've got my recent sticker, my Hubcap sticker. I got my my recent sticker, my Hubcap sticker. I got my Hubcap sticker too. From our buddy SF Eric. Eric, Michael's over. So who did we talk to this week, Wynn?

Talked to Koskay and Andrew from the Jenkins Project, formerly Hudson, about their continuous integration server. Very cool. One of those great tools lets you know who broke the build. Who broke the build. Almost a little like C.I. Joe with GitHub's project. Yeah.

Cool name on that one. Yeah. Knowing is half the battle. Knowing is half the battle. So Jenkins, they went through a rebrand recently, a rename? Yeah. So we talked a little bit about the controversy there and how they kind of parted ways with Oracle

and the corporate backing they had when the project was named Hudson. And now pretty much the whole core team has moved on as the name Jenkins. So we talked about that and a little bit about what you would use Jenkins for. Sweet. Fun episode. Should we get to it? Let's do it. We're chatting today with Koski and Andrew from the Jenkins team. So I'm going to let you guys introduce yourselves and a little bit about what you do with the project.

So Koski, why don't you go first? Okay, so I guess I'm the creator of the original Hudson slash Jenkins project. And I've been involved in it ever since. Andrew? I'm the build guy here at Cloudera and a contributor to Hudson, sorry, Jenkins core and plug-in, and have been for almost two years now. So I bet you we do that a lot on this call, Hudson to Jenkins. So for those that might be confused, who wants to give a little back story

before we jump into what Jenkins actually is around its history as Hudson? Right, okay. So I think I can take that. So, well, I was working for some microsystems for quite some time, I think nine years now. Well, nine years or so. And I'm a guy who just enjoys writing programs. So my day jobs didn't keep me busy enough. So I have all these projects that I'm doing for hobby.

And Hudson was one of those that I started around 2004. And, well, like I said, I have many other projects, but this one, for some reason, got traction, I guess, both inside the company and outside. And so I sort of gradually grew from there. And I think over the years, it attracted a lot of users as well as the developers. And then so we are now a very modest-sized project now.

So I think that's the brief history of the project. How many users do you think you have? Yeah, so we actually have some means. So the Hudson or Jenkins now has a mechanism to sort of ping back our server

because that's how they get updates. And then we also collect anonymous usage statistics. And so that gives us some sense of how much the project is used out there. So I think we haven't tallied the number for a while, but the last time we checked, which was around, I think, the last summer, I think we are estimating about 26K

to 30K installations worldwide. That's fantastic. So given that each one is a server app, I think we have probably about half a million users right there. Yeah, those were real installations that we saw multiple times that had actual projects on them, not just somebody spinning it up locally to take a look at it. Those were real production instances.

So this podcast covers a wide range of listeners. For the uninitiated, explain a little bit about what continuous integration is. To me, I guess the ultimate summary of the CI is like, well, we the human beings want to be lazy. And so the program like CI is really just do everything that the machines can do. And they free us up for the kind of things that only we can do. So that's like the ultimate long-term goal. And I guess, but in the meantime, at the more short term,

what most of the users are using Jenkins for is basically having the servers build their program and test them, and also perhaps deploy to your target environment or do some other peripheral automations. I think reporting on the builds and tests. Oh, yeah. I think reporting on the builds and tests is probably the biggest visibility thing for Jenkins. You can see what happens with the build, you can see what happened in a previous build, you can see how many tests failed this build versus previous builds, etc. Right. It sort of gives more visibility into the current state of the project.

I think what used to happen in Sun a lot is there are people who occasionally, well, if you take vacations and so on, and then often those are the only people who know which branch you're supposed to be committing or whether the tests are supposed to be passing or failing. So when we didn't have Jenkins before,

those information are basically hard-coded into people's brains. And now with applications like CI servers in general, but Jenkins in particular, when you have those data available on the web application, it's much easier for the managers and so on to get some sense of what's going on. Jenkins also offers a lot of features that are useful for management

type of people too, like you have the clouds that you display if a project isn't building well. Yeah. Yeah, we actually took some heat for some of the more visibility stuff. I guess the managers back then didn't really know how horrible our state of the programs are. The engineers kind of got more visibility stuff. I guess the managers back then didn't really know how horrible

our state of the programs are. The engineers kind of got away with that, that there is this stuff that, you know, so when I put, I actually built the, at one point, so one of the things that Jenkins has is this color orb that represents the latest status.

So if it's blue, it's good, and if it's red, it's bad. So at one point, I built this physical device that actually shows it and put that in the coffee status. So if it's blue, it's good, and if it's red, it's bad. So at one point, they built this physical device that actually shows it and put that in the coffee room. And so what happened was that the manager got scared that it's red all the time. My fellow engineers, they didn't like the idea because the manager wasn't supposed to see that.

So can you go over some of the things that makes Jenkins stand out versus the other continuous integration servers that are available? For example, it's written in Java, so you can deploy it anywhere, but people use it for projects that have nothing to do with Java. And can you go over the plugin system? I think that's one of your biggest features right there. Yeah. Andrew, do you want to take that?

Because I think he's been deploying Jenkins in pretty interesting ways. I think he can... Because out of the box, with the plugins you install, you can pretty much do anything that you could ever need 85% of the time, right? Yeah, I think the plugin architecture and ecosystem is really what sets Jenkins apart.

It's so easy to get going writing a plug-in. If you know any Java, you can figure it out pretty quickly. And you can also write them in Ruby. And now there's work on writing them in JRuby as well. So there's so many extension points to build off of so that if you've got a test tool you want to do reports on, a coverage tool, static analysis, there's easy ways to integrate that, either using existing plugins or if you've got something new that's not out there, you can write something yourself very easily

and then publish it through the Jenkins Update Center to get to all the other Jenkins users out there. So that's what pulled me in originally was that I started playing with Jenkins and needed a change to the Quick Case plugin, and next thing I know, I'm the Quick Case plugin maintainer. And then writing more plugins any time there's something else that's not there that I'd like to have there.

And that ease of use and that ease of extensibility is really great as a build hacker, as a tools guy. Being able to improve your tools on the fly is fantastic, especially compared to a large commercial build management server that doesn't really expose its data, that doesn't really encourage you to improve it. It's night and day. Yeah, it's really helped us that our users are also developers.

So when they find some missing things, as long as we provide a means for them to scratch their own,ch they do i think that's really this part of the success of the jenkins is that this plug-in system that allow people to you know just scratch their own little itch and then so over the time i think people have well so i had to spend a lot of time actually to get that infrastructure in place but once that's in place the people showed up in mass and then wrote the plugin. I think we have today about like 300 plugins that's covering not just Java,

but PHP, Ruby, Python,.NET, C++, and what have you. Do we actually have like a COBOL plugin? Somebody was talking about a COBOL plugin, but that kind of terrified me as a concept. I'm just blown away continuously when I'm using it, how easy it is to set anything up. Like I was discussing internally, I was on an IRC channel that we created for my organization. And they were just like, yes, some people use this for reporting builds. And I think it took me two minutes from start to finish to get it to push every test up

report it into the IRC channel it's just amazing I love it. The barrier to get going with pretty much anything is almost nil when you compare that to some of the other CI tools out there the older ones

that have, you've got to go edit files from the server, et cetera. I love it. You just run the warp file, and then everything's pretty obvious and pretty right there. And it supports every version control system there is ever, right? It supports ones that it really shouldn't. Yeah, we support, there is a plugin for visual sourcing.

I don't know why. Last I checked in the last few steps we looked at, I think there were like 10 people who had it installed. That's amazing. I don't know why. I still get the bug reports on that one. So I know people are using it. Well, the biggest bug on that one, of course,

is that it's visual sourcing, but yeah. And so people use it for other things other than just writing tests, right? You can push deploys and everything else with it. You have parameterized builds. Yep. In my previous job at Digg, we used it for pretesting commits, you know, when they get submitted for review, and then an automated deployment and a continuous deployment process such that anytime anything changed

and passed the test, it would keep going through to be tested against other things until it ended up live. So theoretically, the only human intervention you needed was writing the code and reviewing the code. Yeah, anything you can script you can do through Jenkins. Right, so in fact, one of the ways that Jenkins is described is like as a glorified cron. So anything that you execute

at a certain interval, at a certain point or scriptable, you can run it from Jenkins. And it's normally better to do so than doing it from cron. You get the notifications, you can retry at your and it's normally better to do so than doing it from Chrome you get the notifications you can retry at your own choosing if you want to you can script the choreograph multiple things

emails etc etc so Jenkins predates GitHub right? I'm not sure honestly I don't know exactly when GitHub came along So when did you start the Hudson project? That's two so then four Yeah so I think GitHub came along

It lapsed it a couple times Yeah so what has GitHub meant So I'm assuming you added source control Prior to GitHub What has GitHub meant as far as building community And getting community buy-in Around Hudson now Jenkins? Well, pull requests obviously are fantastic.

It's so easy to take this from somebody else rather than having to worry about getting them to put a patch on Jira and then having to apply it and make sure that we've got the right versions against it, et cetera, while with pull requests, somebody can just go fork it, make their change, commit it, push it, and then send in a pull request and we've got what we need. So to me, one of the most important aspects of the project was how to make it easier for other people to come join the development of the most important aspects of the project was how to make it easier for other people

to come join the development of the project. So even when we were using subversion, we had this interesting committer policy that everyone can just become a committer just by asking. Whereas in normal, more mainstream open source project, you normally have to first prove yourself, hanging around long enough and sending in patches before you are accepted.

So we tried various things in an attempt to lower the barrier to entry. And then to me, the Git or the GitHub is like the natural next step to that end. Because then people could just fork a repository and then make some changes. And it makes it easier for us to see those and integrate them back. So when we saw GitHub,

initially I had to say I wasn't quite ready to move the code, but over the time I think we sort of saw the light and then we became an integral part in my mind. So we need to get you guys to tell your buddies about GitHub. We love GitHub, and we would like to cover more Java on the show. The problem is a lot of these code repositories that are popular outside of the GitHub community just make it difficult to peer into the community and see what's hot and not. I'm looking at the top languages on GitHub right now, github.com slash languages,

and Java weighs in at 6% of the projects. So why do you think that is? Yeah, okay. Andrew, go ahead. I think part of that is you've got the really big Apache Java projects all already have their own repositories outside of GitHub historically. So that's some factor to seeing how much of the Java open source ecosystem consists of Apache tools and libraries, etc. But I'm not entirely sure, honestly. It may just be that Ruby and Python

people write too much code. I can attest to that. So speaking of, I'm going to drop a, you guys are still online, I'm going to drop a graphic that you've probably seen in our chat here. How language

programmers view programmers from different languages. I'll drop this in for you, Andy. Paste that into the show notes. You'll have to paste this into the show notes. If you guys remember seeing this. If it's the same thing. Let me take a look.

That's kind of funny. So is that how you see us? Do you see us as the... Well, which one are you, Ruby? Yeah, I'm Ruby, Kenneth's Python. At least you're not PHP. Yeah, Kenneth, I guess you're not represented here. Probably closer to Ruby. No, Python isn't even on there.

That's because it doesn't need to be mentioned. It's just on the whole other plane. Above the fray, as it were, huh? Yes, exactly. Hey, by choice, I'll take Clojure over anything else. Give me my list, and I'm fine. Yeah, we want to give Clojure and Scala on the show as well. So we need to –

Don't those both fall underneath the Java realm though? They do. They do. So we need you guys to hook us up with – They're on the JVM. Oh, okay. Well, I mean I guess you could call it JRuby in that regard as well. But we need you guys to hook us up with the cool projects in these communities

so that we can showcase them on the changelog. Sure. So this whole, you know, you're talking about how Git is fostering the community and, you know, GitHub and Git. Isn't that what triggered this whole Oracle debacle? One of the things, that's one of the triggers, yeah. The community deciding to migrate off of Oracle's infrastructure onto GitHub?

That probably wasn't the underlying cause, but definitely the straw that broke the camel's back, probably. Yeah, back in November, we'd had some very preliminary talks before that point about our infrastructure situation, about eventually wanting to move more to GitHub. I'd already started doing my plugin development with any new plugins in GitHub as of this spring. So, you know, we'd had some talks about that, but, you know, why change was not broken, that sort of thing. And then the Java.net repositories, mailing lists, and the rest of the infrastructure went down while they were moving it all over to new infrastructure and a

new framework, Oracle, moving it without really communicating well with us that that was happening. And so we weren't sure what that meant and we started, you know, trying to make sure we had the source on GitHub so we could keep working, tried to make sure we had Google Groups so that we had a communication mechanism, et cetera. And then there were conflicts stemming from that. Yeah. So then – oh, go ahead. Sorry. No, go ahead. So what's the current status?

I feel like a lot of people are confused because they're calling Jenkins a fork of Hudson when really the Oracle's continued Hudson development is actually a fork of Jenkins, right? That's what I feel. As I've said in a couple posts and emails, that whichever project it is, whatever project it is that Koska is working on, that's the real project to me. I mean, he's too modest to say this himself, but seriously, he's the project. He's written like 85% of the code of core. I mean, he's done remarkable work here. And I can't see a situation where the majority of the community says we want to go rename to Jenkins and Koskay as part of that, where I can't see that being the fork versus the one that's kept the name

but that's about it I should also point out that I guess we did the voting to get the feeling and get to see where the community how the community feels

and so the result of that was likely more than 90% of people were favorable. 214 voters, available voters, people who had been on the mailing list before the vote started, voted to rename to Jenkins. 14 voted to stay with the status quo under Oracle Central. Right. So I guess my argument is that, well, if the 93% of, you know, well, let's say, well, the people are moving with, you know, moving us to Jenkins, you know, well, and they call that a fork.

I don't know. I don't know how you, I think that's not a fair description. Is there going to be any code sharing between the two projects? We'll see. I don't... I honestly... We, in our talks with Oracle, when this actually ended up happening,

they made it very clear that renaming was not saying we want nothing to do with Oracle. It was just saying that want nothing to do with Oracle. It was just saying that because Oracle claims a trademark on the name Hudson, there were restrictions that that put on the project. We didn't have a guarantee that we'd be able to use the name regardless. We weren't able to be a truly independent project while we were beholden to Oracle for the rights to our name. So I felt that, and Koski agreed and apparently most of the community agreed, that we needed to change the name so that we could be an independent project.

That was not saying that Oracle shouldn't be involved. We still offered the third seat on the interim board, governance board, to Oracle. I really wish Oracle had wanted to and been willing to work with the Jenkins community, but that's not what they opted for. They opted to, I guess, Let's see if there's... You know, what code is shared. I don't know what all they'll end up doing. I don't know what all they'll be able to take from us and vice versa based on licenses or copyright or community licensing agreements, et cetera.

Hopefully we can stay compatible as long as possible. Hopefully they, you never know, maybe they'll be willing to play ball eventually. We're not trying to push them away. We're trying to just make sure that it's a healthy, stable, vibrant, independent project. So Matthew McCullough on Twitter, you've answered part of his question, wants to know if there will be any effort to migrate existing plug-ins from using the Hudson name to the Jenkins name. Yes. But it's probably going to be piecemeal.

It'll probably be when there's a reason to release a new version of the project, of a plug-in, besides just changing the name, then do a new release. I know that for my plug-ins, I'm at least looking into that, but it's not... Functionality matters more than cosmetics on a plug-in level, I think. looking into that, but it's not. Functionality matters more than cosmetics on a plug-in level, I think. As long as it says Jenkins at the top banner,

you can tell that you're on Jenkins. I think that it's permissible if one plug-in doesn't quite get all the names exactly right right away. We don't want to make everybody have to reinstall new versions of their plugins just to get a purely cosmetic name change. But over time, I'm sure that will more and more

as plugins get modified, as new releases come out, that the Hudson name will fade out from the plugins. Is that going to affect it, or do you know how that's going to affect the Ubuntu package by any chance, or the Debian packages, I should say? Okay.

So I believe, so let's see. So I believe we only have, the packages we produce are already properly renamed, and I don't think we are in any other sort of official winter repositories. Oh, yeah. Oh, sorry. Well, I have the app source pointed to the Hudson URL.

Is that automatically over at Jenkins now? Or do I need to update that? I believe that Kosuke and others have written up a wiki entry on upgrading Hudson to Jenkins. We can send that to you guys if you'd like. So you think that the migration for everyone should be painless and really wouldn't make any difference at all, right? It should.

Yeah. So far, the reports we got from people are very positive that they were able to smoothly migrate to a newer version. So you just have to say, if you're using the Ubuntu, I guess you just have to say sudo apt-get install jenkins, and then that's it. Sounds good. So this whole debacle kind of brings forth the two-edged sword of corporations backing open source software. Do you have any other comments or opinions on what people should be doing and shouldn't be doing

in terms of that and how to handle certain situations? Because this has got a lot of press and a lot of coverage. It's kind of a hot topic at the moment. Yeah, I don't know what the lesson should be. Is there anything you guys would have done differently from the start? Obviously, you you guys would have done differently from the start or you wish that obviously you wish Oracle would have been a little more willing to to discuss things and you know be part of the board now that you've changed the name right

for instance on jQuery John Resick you know made efforts at some point in the project to move trademarks and licenses and things outside of his personal control into a foundation control, as I understand it. Is that important to do once a project hits a certain critical mass to make sure that no single organization other than that organization that really is looking after the community

can gain control of it? I think that's true. I think it's different depending on the situation. There's plenty of open source projects that are 90% developed at one company, but they share it with the world. And in those cases, I mean, that's, you know, the choice of the developers, the choice of the development and user community. But if it's a project that's not just tied to one corporation, not just tied to one entity,

I think it is important to make sure that it is truly independent, that you don't have one figure playing a bigger, having more disproportionate power because of trademarks and copyrights over the rest of the community, assuming that's the direction you want your project to go. Again, this is, it's entirely up to the project what they want to do. What works for Jenkins doesn't necessarily work for a different project.

There's no one-size-fits-all solution. Yeah, but I think we've come out stronger from this than before. I think it was actually a good thing that this gave us the motivation to look more into more structures and the governance and making sure that the various companies are involved and so on. So that sort of makes helps make the project more independent and I think in the long term that helps.

Even though in the short term it might take some hit. I think it's in the long term I think we'll come out stronger from this than before. We've definitely seen an increase in users and developers wanting to help, wanting to contribute, both bug fixes, help with the rename process, infrastructural matters. It's been heartening.

It's been really nice to get that support, to get more people playing bigger roles, so that everybody's got more of a sense of ownership of the project, because it is the communities. So my wife and I are considering baby names, and I've got to tell you, Hudson was on the list, but I can't say that Jenkins will make the cut. Where did that name come from, and was this also a community vote? So Hudson, the name Hudson was for Butler from Upstairs Downstairs, if I remember correctly, the BBC TV show. Is that right? That's right, isn't it? No, that's not. So I guess the origin of the name is that

as I mentioned, I think of this software as really as one more person to your development team. I liked very much the idea of giving it the name of the person so that you can say, well, let's have Hudson look at this, or let's have Jenkins already did this so let's have the Jenkins

already did that or something like that so you know so the so I thought well this is a program that helps other people so well what what kind of people help other people but butlers do and so so hence the logo of that little gentleman so he's a but. And then the Hudson just sounded like a British butler name to me. It sounds like, you know, when I asked the people here, it apparently isn't, but somehow I got the impression that the Hudson was like a British sounding name. So then, you know, then, so that was like 2004. And then recently we had to come up with something else. So we actually had, you know, we looked at a few other names. I think our primary choice was actually Alfred, you know, the famous popular quote.

Yeah. But we, fairly late into the game on that one, we discovered an application for the Mac. Yeah, I was going to mention that. Called Alfred. How we didn't notice that until then, I'm not entirely sure. But, yeah, so then we had to come up with something else that would still evoke the Butler feel,

something that would still fit with the theme and the conception. And Jenkins, I think, is a pretty good option. I think it's a... We didn't put the specific name up for a vote just because then we'd end up with 70,000 suggestions. Was Niles up for a vote? Security was sort of embraced it. Yeah, people had all kinds of suggestions

when this whole thing had started. I can imagine. Someone suggested that we call it RARI in honor of the CEO. So this is where we turn the show upside down a little bit and we've added another question or two to this kind of ending salvo so i'll put you guys on the spot first question and i'll hit you first koski who's

your programming hero ah okay so so that's easy question so there's a guy called james clark um he's um i think he now lives in thailand but yeah, he's in a very unbelievable position of being very rich and very smart. So like, he doesn't have to do work and he just want to do

things that he wants to do. And he's been, so he's like so smart that I actually met him a few times and his brain is actually bigger. You can see that

like his head above the eyes is like actually swelling to accommodate his brain size and so he's been my hero ever since um he you know when i looked at his color it's just just amazing and um so that's the height that i'm trying to get to but you know there are sometimes you just see someone that's so good that you kind of get depressing because you see the chasm that you can't cross. You'll see. At least I'm younger than him, so if I survive him longer, that might be a way. Andrew? Honestly, I have a hard time imagining anybody that much better

than Kosuke, who is definitely one of my programming heroes, just for the sheer volume of amazing code that he's written. Also, Guy Steele and other Lisp hackers, just aesthetically, I love Lisp, and I love thinking about Lisp, and so guys have done really great work in that area, and in language design. Impressed the hell out of me. Yeah, you have to think with Lisp that I've have done really great work in that area and in language design. Impressed the hell out of me. Yeah, you have the thing with Lisp that I've been always curious about. My dad was an MIT grad, so I grew up with the little Lisper in the room with the computer. And then in college, we did our second year of CS courses in Scheme, and there's just something elegant about Lisp languages and about thinking about programming in that functional way, where code is data and data is code.

I'm not very good at it, but it really helps me when I run into a programming challenge to think about the problem and how one would solve it from that perspective. It tends to help me come up with less buggy solutions, if nothing else. So you grew up within an environment surrounded with a sea of parentheses? Yeah, basically. And I've been on Emacs for 14 years.

Oh, so Lisp Emacs Which Emacs Lisp Which Lisp is the Kosher implementation Most of the Lisp I've done in The last years have been just programming Challenge stuff and Clojure

I like to dabble in Go ahead sorry Sorry go ahead I like to dabble in MZ scheme every once in a while, but I still haven't wrapped my head completely around the concept. It's like an exercise for me. I could never be productive in that environment. Definitely take a look at Structured Interpretation of Computer Programs, the old intro textbook from MIT. It's a brilliant resource for, A, understanding programming, and, B, understanding schema-less languages.

I treasure my copy. So outside of Jenkins, what software gets you guys excited that you really want to play with in the future of the whole programming landscape. Interesting question. Well, I'm a build guy. So build tools really fascinate me. I love Maven for Java builds.

I think Selenium is absolutely fantastic. And I really enjoy seeing this sprawl of languages running on JVM besides Java. I think that

the ability to write in so many different languages, but share code between them, I think, is really great. So if you had a completely open weekend this weekend, and you weren't allowed to touch anything related to build servers, what project

would you play with? I have absolutely no idea. I've been trying to set up this, I guess, the home audio automation. So I bought the Airport Express. I guess that's how they call their wireless router at Apple. They got some DRM to protect it down, but they got the streaming protocol that can send audio over there.

So I was wondering if I could hack that a little bit so that I can get my speaker hooked up there to receive audio from my computers. That would be very cool. The problem is that everything like that eventually comes back to Jenkins for me. I had these lots of holy crocs, but one way or the other

they come back to Jenkins. So in the context of Jenkins it would be like if the build would break, you know, this is sort of, you don't normally have a speaker, but with this audio over the internet, you could actually send the, you know, send the audio over that and then you get,

so I thought that would be funny. I haven't actually done that, but. Well, thanks for taking the time, guys. We really appreciate you telling us the backstory of Jenkins and A. Hudson. And hopefully it'll just keep on going as far as the momentum that you've seen so far. Yep. Thank you. See it in my eyes So how could I forget when

I found myself for the first time Safe in your arms As the dark passion shines