Screaming in the Cloud - Automating in Pre-Container Times with Michael DeHaan

Episode Date: May 5, 2022

About MichaelMichael is the creator of IT automation platforms Cobbler and Ansible, the latter allegedly used by ~60% of the Fortune 500, and at one time one of the top 10 contributed to projects on GitHub.Links Referenced:Speaking Tech: https://michaeldehaan.substack.com/michaeldehaan.net: https://michaeldehaan.netTwitter: https://twitter.com/laserllama

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and it's spelled R-E-V-E-L-O.

It means I reveal. Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Ravello has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to basically those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes but isn't limited to talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability as well as, you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States.

So you can hire full time remote engineers who share most of the workday as your team. It's an end-to-end talent service. So you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io slash screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming. get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly

this. To learn more, visit launchdarkly.com and tell them Corey sent you and watch for the wince. Once upon a time, Docker came out and changed an entire industry forever. But believe it or not, for many of you, this predates your involvement in the space. There was a time where we had to manage computer systems ourselves with our hands, kind of, like in the prehistoric days, chiseling bits onto disk and whatnot. It was an area crying out for automation as we started using more and more computers to run various websites. Ooh, that's a big website. It needs three servers now, et cetera. The times have changed rather significantly. One of the formative voices in that era was Michael DeHaan, who's joining me today.

Originally one of the, or if not the creator of Cobbler and later for which you became better known Ansible. First, thanks for joining me. Thank you for having me. You're also making me feel very, very old there. So, uh, yes, I hear you. I, I keep telling people I'm in my mid thirties and my wife gets incensed because I'm turning 40 in July. But still, I go for the idea of, yeah, the middle is expanding all the time, but it's always disturbing talking to people who are in our sector who are younger than some of the code that we're using, which is just bizarre to me. We're all standing on the backs of giants, like it or not. One of them's you. Well, thank you. Thank you very much. Yeah,

I was talking to some undergrads. I was doing a little bit of stuff helping out my alma mater for a little bit and teaching somebody a rest lecture i was like in another year rest is going to be older than everybody in the room and then i was just kind of uh scared yeah it's been a wild ride for basically everyone who's been around long enough you don't fall off the teeter-totter and wind up breaking a limb somewhere. So back in the battle days, before cloud, when everything was no longer, things back then weren't constrained by how much room you had on your credit card, like they are today with cloud, but instead by things like how much space you had in the data center, what kind of purchase order you could ram through your various accounting departments.

And one of the big problems you have is, great, so finally, never on time, Dell has shipped out a whole bunch of servers or HP or Supermicro or whoever, and the remote hands, which is always distinct from smart hands, which says something very insulting, but they seem to be good about it, would put them into racks for you. And great, So you'd walk in and see all of these brand new servers with nothing on them. How do we go ahead and configure these things? And by hand is how most of us started. And that means, oh, great, we're going to screw things up and not do them all quite the same. And it's just a treasure and a joy. Cobbler was something that you came up with that revolutionized how provisioning of bare metal systems worked. Tell me about it. Yeah, so it's basically just glue.

So the story of how I came up with that is I was working for the emerging technologies group at Red Hat, and I just joined. And they were like, we have to have a solution to install Xen and KVM virtual machines. So obviously, everybody's familiar with EC2 and things now. But this was about people running non-VMware virtualization themselves. So that was part of the problem. But in order to make that interesting, we really needed to have some automation around bare metal installs, and that's PixieBoot. So it's TFTP and DHCP protocol and all that kind of boring stuff. And there was glue that existed, but it was usually humans would have to click on buttons

to Red Hat had system config net boot. But what really happened was system admins all wrote their own automation, like every single company. And the idea that I had, and it was sort of cemented by the fact that my boss, a really good guy, left for another company and I didn't have a boss for a couple of years. I i was like i'm just going to make irc my boss and let's get all these admins together and build a tool we can share right so that was a really good experience and it's just basically gluing all that stuff together to fully automate and install over a network so that when a system comes on you can either pick it out from a menu or maybe you've already got the MAC address.

And you can just say, when you see this MAC address, go install this operating system. And there's a kickstart file or a pre-seed in the case of Debian that says, when you're booting up through the installer, basically, here's just the answers and go do these things. And that install process is a lot slower than what we're used to. But for a bare metal machine, that's a pretty good way to do it. Yeah, it got to a point where you could walk through and just turn on all the servers in a rack and go out to lunch, come back, they would all be configured and ready to go. And it sounds relatively basic the way we're talking about it now, but there were some gnarly edge cases. Like, when I rebooted the database server, why did it wipe itself and reprovision? And it's, oh dear. And you have to make sure that there's a safety built into these things. And you also don't want

to have to wind up plugging in keyboard and monitor to all of these individual machines one by one to hit yes and acknowledge the thing. And it was a colossal pain in the ass. That's one of the things that cloud has freed us from. Yeah, definitely. And one of the nice things about the whole cloud environment is if you want to experiment with those ideas, like I want to set up some DHT or DNS, I don't have to have this massive lab and all the electricity and costs. But if I want to play with a load balancer, I can just get one. And that kind of gives the experience of playing with all these data center technologies to everybody, which is pretty cool. On some level, you can almost view the history of all these things as a speeding things up. With a well-tuned Gobbler install, it still took multiple minutes, in some cases tens of minutes,

to go from machine you're powering on to getting it provisioned and ready to go. Virtual machines drop that down to minutes, and cloud, of course, accelerated that a bit. But then you wind up with things like Docker, and it gets down to less than a second. It's the mean time to dopamine. But in between the world of containers and bare metal, there was another project, again, the one you're best known for, Ansible. Tell me about that, because I have opinions on this whole space. Yeah, so how Ansible got started? Well, I guess configuration management is pretty old. So the people writing their own scripts, CFEngine came out, Puppet was a much better CFEngine. I was working at a company and I kind of wanted another open source project

because I enjoyed the cobbler experience. So I started Ansible on the side, kind of based on some frustrations around Puppet, but also the desire to unify Capistrano kind of logic, which was like, how do I push out my apps onto these servers that are already running with Puppet-style logic? Is this computer's firewall configured directly? And is the time set correctly? And you can obviously use that to install apps, but there are some places where that blurred together, where a lot of people are using two different tools. And there's some prior art that I worked on called Funk, which I wrote with Seth Vidal and Adrian Likens at Red Hat, which was like 50% of the Ansible idea.

And we just never built the config management layer on top. So the idea was make something really, really simple that just uses SSH, which was controversial at the time because people thought it wouldn't scale because I was having trouble with setting up Puppet security because it had DNS or timing issues or whatever. Yeah. Well, let's dive in a bit to what config management is first, because it turns out that not everyone was living in the trenches in quite the same way that we were. I was a traveling trainer for Puppet for a summer once, and the best descriptor I found to explain it to people

who were not in this space was, all right, let's say that you go and you buy a new computer. What do you do? Well, you're going to install the applications you like to use. You're going to set up your own user account. You're going to set your password correctly. You're going to set up preferences, copy they all need to be the same. How do you do that? Well, that is the world of configuration management. And there was sort of a bifurcation there, where there was the idea of, first, we're going to have configuration management that just describes what the system should look like, and that's going to run on a schedule or whatnot. And then you're going to have the other side of it, which is the idea of remote execution, of I want to run an arbitrary command on this server or this set of servers or all the servers, depending upon what it is. And depending on where you started on the side of that world, you wound up wanting things

from the other side of that space. With Puppet, for example, it was very oriented configuration management. And the question became, well, can you use this for remote execution with arbitrary commands? And they wound up doing some work with mCollective, which was a very complicated and expensive way to say, no, not really. There was a need for things that needed to hang out in that space. The two that really stuck out from that era were Ansible, which had its wild runaway success, and the one that I was smacking around for a bit, SaltStack, which never saw anywhere approaching that level of popularity. Yeah, sure. I mean, I think that you hit it pretty much exactly right. And it's hard to say what makes certain things take off. But I think like the just SSH approach was was interesting because well, for one, everybody's running it. But there was this belief that this was not scale.

And I tried to optimize the heck out of that because I like performance. But it turns out like that that wasn't really a business problem because if you can imagine you just wrote this little bit of automation and you're going to run it against your entire infrastructure and you've got 30,000 machines. Do you want that to, if you were to run an update command on 30,000 machines at once, you're going to DDoS something. Definitely.

Yeah, suddenly you have 30,000 machines all talking to the same things at the same times. You want to do them in batches or smear it across. So because that was there, you just add batch support in Ansible and things are fine, right? People want to target little small groups of things. So that whole story wasn't true. And I think it was just a matter of testing this belief that everybody thought that we needed to have this whole network of things. And honestly, Salt's idea of using a message bus is great, but we took a little bit different approach with the YAML because we had YAML with variables in it, but they had something that

compiled down to YAML. And I think those are some differences in the dialect and some things other people preferred. And they used Jinja at one point to wind up making it effectively Turing complete. You could wind up having this ridiculous loop flow control and loops and the preferred. And they used Jinja at one point to wind up making it effectively Turing-complete. You could wind up having this ridiculous like loop flow control and loops and the rest. And it was an interesting exposure to things, but yikes at the same time. If you use all the language features in anything, you can make something complicated and too complicated.

And I always like, I wanted automation to look like grocery lists. And when I started out, I said, hey, if anybody's doing this all day for a day job, I will have failed. And it clearly shows you that I have because there are people that are doing that all day. And the goal was, let me concentrate on dev and ops and my other things and keep this really, really simple. And some people just like get really, really into that automation technology, which is, in my opinion, why some of the earlier stuff was really popular.

Because systems were bored, so they see something new and it's kind of like a Java developer finding Perl for the first time. And they're like, I'm going to use all these things. And they have all their little widgets and it gets like really complicated. The thing that I always found interesting and terrifying at the same time about Ansible was the fact that you did ride on top of SSH, which is great because every company already had a way of controlling access via SSH to its systems. Everyone uses it, so it has an awful lot of eyes on the security protocol and the rest. The thing that I found terrifying in the early days

was that more or less every ops person would wind up checking this out onto their laptop or whatnot. So whenever they wanted to run something, they would just run it from their laptop over a VPN or whatnot from wherever they happened to be, and you wound up with a dueling banjos type of circumstance where people were often not doing it from a centralized place. And in time, best practices emerged where, okay, that is going to be the command and control server where that runs that and you log into it. And then

you start guarding that with CI, CD flows and the rest. Like anything else, it wound up building some operational approaches to it. Yeah, I kind of think that created a problem that allowed us to sell a product, right? Which was good. If you knew what you were doing, you could use Jenkins completely and you'd be fine, right? If you had some level of discipline

and access control and you wanted to wire that up. And if you think about cloud, this whole like shadow IT idea of, I just want to do this thing there where i'm going to get an amazon account it's kind of the same thing it's like i want to use this config management but it's not approved who can stop me right and that that kind of probably got us in the door in a few accounts that way but yeah i did definitely create the problem where

multiple people could be running things at the same time so yeah i mean that's true and the's true. And the idea of, hey, maybe I should be controlling these things in Git or some other form of version control was sort of one of those evolutionary ideas that, oh, we could treat this like code. In the early days of DevOps, that was a controversial thing. These days, you say you're not doing it, and people look at you very strangely. And things were going reasonably well in that direction for a while, and then this whole Docker thing showed up where, well, what if instead of having these

long-lived servers where you have to install updates and run patches and maintain a whole user list on them, instead you had this immutable infrastructure that every time there was a change, you would just go ahead and deploy a brand new set of servers. And you could do

this in the olden days with virtual machines and whatnot. It just took a long time to push things out. So do I really want to roll the entire fleet for a two-line config change? Probably not. So we're going to batch it up or maybe do this hybrid model. With Docker, it takes less than a second to wind up provisioning that switching over to the new container series and you're done. You can keep going with that. That really solved a lot of these problems. But there were companies that, like the entire configuration management space, who suddenly found themselves in a really weird position. Some of them tried to fight the tide forever and say, oh, this is terrible because it means we don't have a business model anymore.

But you can only fight the future for so long. And I think today we'd be hard-pressed to say that Docker hasn't won on some level. Deploying the undercloud because people are still running essentially clouds on top of clouds to get their Kubernetes deployments going. And those are monstrous. Or maybe to deploy a data layer. I know Kafka has gotten off of Zookeeper, but the Kafka Zookeeper thing, and I don't remember if Zookeeper requires etcd or not. But managing those sort of long, persistent implications, it still has a little bit of a place where it exists. But I mean, I think the whole immutable systems idea is theoretically completely great.

I never was really happy with the whole Docker development workflow. And I think it does create a problem where people don't know what they're deploying and you kind of encourage that to where they could be deploying different versions of libraries. And that's kind of just a problem

with the whole microservices thing in general, where did somebody change this? And then I was working very briefly at one company where we essentially built a whole dashboard to detect service versions and what version of the base image everybody was on and all these other things. And it can get out of hand too.

So it's kind of like trading some problems for other problems, I think to me. But in general, containerization is good. I just wish the management glue around it was easy, right? I wound up giving a talk at a conference a while back, 2015 or so, called Heresy in the Church of Docker. And it was a throwaway five-minute lightning talk, and someone approached me afterwards with,

hey, can you give the full version of that at ContainerCon? I was like, there's a full version? Yes, yes, I can. And it talked about a number of problems with the management layer and the rest. Now, Kubernetes absolutely solves virtually every problem that I identified with it. But when you look at the other side of it, getting Kubernetes rolled out is effectively, you get to cosplay being a cloud provider yourself. It is incredibly complicated. And of course, we're right back to managing it all with YAML. Right. And I think that's an interesting point, too, is I don't know who's exactly responsible for the YAML explosion. And I like it as a data format. It's really good for

humans. Cobbler originally used it more of an internal storage, which I think was a mistake because I was trying to avoid setting up a database at the time. Because I knew if I had to require setting up a database in 2007 or 2008, I'd get way less users, so I'd use flat files. A lot of the YAML dialects people are developing now are very, very nested, and it requires loading a web page for the docs all the time and reading what's valid here, what's valid there. I think people learned the wrong lesson from Ansible's YAML usage. It was supposed to be like, YAML usage, right?

It was supposed to be like, YAML is good for things that are grocery lists. And there's a lot of places where I didn't do a good job. But when you see things taking methods, taking 15 parameters, and you have to constantly have the reference up, maybe that's a sign that you should do something else. At least you saved us on some level from having to do this all in XML. But still, there are wrong ways and more wrong ways to do it. I don't think anyone can ever agree on the right way to approach these things. Yeah, I mean, YAML at the time was a good answer because I knew I

didn't want to write and maintain a parser as like a guy that was running a project. We had a lot of awesome contributors. But if I had to also maintain a DSL, not only does that mean that I have to write the code for this thing, which I observed slowing down some other projects, but also that I'd have to explain it to people. Looking kind of like Bash was not a bad thing. Not having to know and learn something. So you kind of feel really effective in about 15 minutes or something like that. One of the things that I find really interesting about you personally is that you were starting off in a bare metal world.

Ansible was sort of wherever you wanted to run it. Great, as long as there are systems that can receive these things, we're great. And now the world has changed. And for better or worse, configuration management slash remote execution is not the problem it once was. And it is not a best practice way of solving a lot of those problems either. But you aren't spending your time basically remembering the glory years. You're actively moving forward doing some fairly interesting stuff. How would you describe what

you're into these days? I tried to create a few projects to kind of build other systems manager things for the same audience for a while. I was building a build server and trying to do some next-gen config stuff. And I saw people weren't interested, but I like having conversations with people. And I think one of the lessons from Ansible was how to explain highly technical things to technical audiences and cut out a lot of the marketing goo and all that.

How to get people excited about an idea and make a community be really authentic. So I've been writing about that for, really, it's rebooted blogs, only a couple weeks old. But also kind of trying to do some helping out companies with some basic marketing kind of stuff. There's just this pattern that everybody has where every website starts with this little basic slogan, two buttons, and then there's a bunch of adjectives, but it doesn't say anything. So how can you have really good documentation and how can you explain an idea? Because really the reason you're in it is not just to sell stuff, but it's to help people and to see them get excited about your ideas.

And we're not doing a good job in this world where there's thousands upon thousands of applications all competing at once. How do you rise above that? And that's always the hard part is at some point, the systems become your identity and you become known for a thing. And when you start branching out from that thing, you bring the expertise from that area that you were in, but you start applying it to new things. I feel like so many companies get focused and people get focused on assuming

that their audience is just like them, where they're coming in with the exact same biases, the exact same experiences. And given that basically no one was as deep in the weeds as you were when it came to configuration management, that meant that you were spending time in that side of the world, not in other pursuits, which aligned in some ways more directly with people developing other things. So I suspect this might be one of the weird things we have in common when we show up and see something new and a company is really excited. It's like, it's basically a few people talking together. Like both founders are technical and they're super excited about something they can't quite articulate. And it's this slow down, tell me exactly what it is your product

does. And that's a hard thing to do because my default response was always that if I don't understand that it's clearly the way in which I am deficient somehow, but marketing is really about clear communication. And there's not that much of it in our space, at least not for early stage companies. Yeah. I don't know why that is. I mean, I think there's this belief in that there's like this buyer audience where there's some vice president that's going to buy your stuff if you drop the right buzzwords. And 15 years ago, you had to say Synergy, and now you say Time to Value or Total Cost of Ownership or something. And I don't think that's true. I mean, I think people use products that they like and that they need to be shown them to try them out.

So why can't your webpage have a diagram and a screenshot instead of this picture of a couple people drinking coffee around a computer, right? It's basic stuff. But I agree with you, I kind of feel dumb when I'm looking at all these tech products that I should be excited about. And like the way that we get there is we ask questions. And the way that I've actually figured out what some of these things do is usually having to ask questions from someone who uses them that I randomly find on my diminishing circle of friends, right? And that's kind of busted. So Ansible definitely had a lot of privilege in the way that it was launched

in the sense that I launched it off Cobbler List. And Cobbler List started off of ET Management Tools, which was a company list. But people can do things like meetup groups really easily. They can give talks. They can get their blogs reblogged and hope for some hacker news or Reddit juice or whatever. But in order to get that to happen,

you have to be able to talk to engineers that really want to know what you're doing and they should be excited about it. So learn to talk. You have to speak their language, but without going so deep in the weeds that the only people that understand it are the folks who are never going to use your product

because they want to build it themselves. It's a delicate balance to strike. And it's a difficult thing to do too when you know about it. So when I was developing all all the ansible docs i've told people many times and i hope it's true that i like spent like 40 of my time just on the website and the docs and whenever i heard everybody complain i try to fix it but the idea was like you can lose somebody really fast but you kind of have to forget what you know about the product so the worst person to sometimes look at that is the person that built it so you have to forget what you know and the product. So the worst person to sometimes look at that is the

person that built it. So you have to forget what you know and try to see like what questions they're asking. What do they need to find out? How do they want to learn something? And for me, I want to see a lot of pictures. A lot of people write a bunch of giant walls of text or worse for me is when there's just these little pithy expressions and I don't know what they mean, right? And everybody's like kind of doing that these days. This episode is sponsored in part by our friends at Chaos Search. You could run Elasticsearch or Elastic Cloud or OpenSearch as they're calling it now

or a self-hosted Elk stack. But why? Chaos Search gives you the same API you've come to know and tolerate along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch.

They're also available now in the AWS Marketplace. If you'd prefer not to go direct and have half of whatever you pay them count toward your EDP commitment, discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again. One thing that I've really found myself enjoying recently has been your Substack-based newsletter. Speaking Tech is what you call it. And I didn't quite know what to expect when I signed up for it, but it's been a few weeks now. And you are more or less hitting across the board on a bunch of different things, ranging from engineering design patterns to a teardown of random companies' entire website from a marketing and messaging perspective,

which I just adore personally. That is very aligned with how I see the world. Where's that coming? Yeah, to hiring a bunch of other stuff. Let's talk about, for example, the idea of those teardowns. I always found that I have to be somewhat careful in how I talk about it when I'm doing a tweet thread of something like that, because you are talking about people's work, let's be clear here. And I tend to be a lot kinder to small, early-stage companies than I am to, you know, $1.6 trillion companies who really should have solved for this by now on some level. But so much of it misses the mark of, great, here's the way that I think about these things. Here's the way that I don't understand what the hell you're telling me. An easy example, this for me at least,

I'm curious to get your thoughts on it. I tend to almost always just skim what they're saying. Great. Let's look at the pricing page because I find that that speaks to people in a way that very often companies forget that they're speaking to customers. Yeah, for sure. I always try to find the product page lately. And then the product page now is like a regurgitation of the homepage. But what you said earlier, I think I try to stay nice to everybody, but it's good to show people how to understand things by counterexample to some extent, right? And like, oh, I've got some stuff coming out. I don't know when this is actually going to get published. But next week where I was like just taking random snippets of homepages and like, what's everybody doing with the header these days? And there's just like ridiculous amounts of copying and going on. But it's not just for like people with companies because everybody listening here isn't going to have a company. If you have a project and you wanted to get it noticed, right? I think like in

the early days, the projects that I paid attention to and got excited about were often the ones that spent time on their website and their messaging and their experience. So everybody kind of understands you have to write a good readme now. But something like the early Ruby crowd, for instance, did awesome, awesome web pages. They know how to pick out fonts and I still don't know how to pick out fonts. I ask someone good at those things. That's how I pick them. Yeah, yeah. It's not my job. Get somebody that's good at that. But all that matters, right? So if you do invest a little bit in not promoting yourself, not promoting your company, but trying to help people and

communicate to them, you can build that audience around your thing and it makes it a lot more interesting. There's so many great tools out there that I find on GitHub that other people have to either point me to or I find it when I'm looking at it from a code first perspective, just trying to find a particular example of a library being used, where they do such a terrible job of describing the problem that they solve. And it doesn't take much. It takes a paragraph or two at most. Or the idea that, oh yeah, here's a way to do this thing. Just go ahead and get your credential file somewhere else. Great. Could you maybe just link to an example of how to do that? It's the basic stuff. Assume that someone who isn't you might possibly want to use this.

And I'm not even slightly suggesting that you wind up talking your way through how to do all of that. Just link to somewhere that has a good write-up of it and call it good. Just don't get in the way of people's first-time user experiences. Yeah, for sure. For some reason, that's a radical thought. Yeah, and I think one of the things the industry hasn't, well, not the industry, it's not their problem to solve. But like, we don't really have a way for people to find what's cool and interesting anymore. So various people have their own little lists on GitHub or whatever. But there's just so many

people posting on the one or two forums people read, and it goes by in a day. So it's really, really hard to get attention. Even your own circle of followers isn't really logging into Twitter or anything or LinkedIn or there's all the congratulations for your five years at Acme Corp kind of posts. And it's really, really hard to get attention. I feel for everybody. So like, if somebody like GitHub or Microsoft is listening, and you wanted to build like a dashboard of here's the cool 15 projects of the week kind of thing where everybody would see it and start spotlight some of these really cool new things. That would be awesome, right? Whenever you see those roundups, though, it's things like Kubernetes and Docker and great. I don't think those projects need the help in the same way. No, no, they don't. It's like maybe somebody's cool data thing or a cool visualization or the other thing that's completely random. But I used to write fun graphics programs for fun or games and libraries,

and I don't see that anymore, right? Maybe if you find it, you can look for it. But the things that get people excited about programming, maybe they have no commercial value at all, but the way that people discover stuff is getting so consolidated. It is about Docker and Kubernetes, and everyone's talking about these three things.

And if you're not Google or you're not Facebook, or Amazon, obviously, it's hard to get attention. Open source on some level has changed from a community perspective. And part of it is because once upon a time, you could start with the very low-level stuff and build something, get it up and working. And that's where things like Comble and Ansible came out of. Now it's click the button and use the thing everyone else is using. And if you're not doing that, what are you doing over there? So the idea of getting started tinkering with computers are built on top of so many frameworks and other things. That's always been the case, but now it's much more apparent in some ways. Okay, I'm going to go ahead and build out my first HTML file and serve it out using something

in Node. Great. Well, what is those NPM stuff that's scrolling past? The devil. That is the devil's own language you are seeing scroll past, and you don't need to worry about that. Just pretend it's not there. But back when I was learning how this stuff worked, paying attention to things scrolling past, like, you know, compilation messages and the Linux boot story as it wound up scrolling past. Terrible story. The protagonist was unrelatable, but all right. And you start learning how these things work when you start scratching at the things that you're just sort of hand-waving and glossing over. These days, it feels like every time I use a modern project, that's everything.

I mean, it is. And like what React has like 2000 dependencies, right? So how do you ever feel like you understand it? Or when recruiters are asking for 10 years in Amazon, and then or we find a library, then it can only explain itself by being like this other library and requiring these other five and you read one of those and it becomes like this tree of knowledge that you have no way of possibly understanding. So we've just built these stacks upon stacks upon stacks of things. And I tend to think I kind of believe in minimalism. And wouldn't it be cool if we just burned this all and start, we burned the forest and let something new regrow.

But we tend to not do that. We just, now we're running a cloud on top of a cloud and our JavaScript is thousands of miles high. I really wish that there were better paths for getting started. I used to think that the right way to wind up learning all this stuff works is to do what I did. Start off as the grumpy sysadmin type or help desk and then work your way up and the rest.

Those jobs aren't there anymore, and it doesn't lead people in a productive environment. Oh, you want to build a computer game? Great, for an iPhone. Terrific. Where do you go to get started on that? It's a hard thing to do. And people don't care at that scale, nor should they necessarily, on how to run your own servers. Back in the day when you wanted to have a blog on the internet,

you were either reduced to using LiveJournal or MySpace, or you were running your own web server and had to learn how to make sure that it didn't become an attack platform. There was a learning curve that was fairly steep. Now there are so many different paths to go down, you don't really need to know how any of these things even work. Yeah, I think like one of the, I don't know whether DevOps means anything as a topic or not, but one of the original pieces around that movement was systems administrators learning to code things and really starting to enjoy it, whether that was Python or Ruby and so on. And now it feels like we're gluing all things together,

but that's happening in app dev as well, right? The number of people that can build a really, really good library from the ground up, like something that has C bindings, that's a really, really small crowd. And most of it, what we're doing is gluing together other people's libraries and compensating for the flaws and bugs in them and duct tape and error handling or whatever. And it feels like programming has changed a lot because of this. And it's good if you want to get an idea out quickly, no doubt,

but it's a different experience. The problem I always ran into was the similar problems I had with doing Debian packaging. It was always the, oh great, there's going to be four or five different guides on how to do it. Same story with RPM. And they're all going to be assuming different things, and you can cross over between them without realizing it. And then you just do something monstrous that kind of works until an actual Debian developer shoves you aside like you are a hazard to everyone around you. Let me do it for you. And there we go. It's basically get people to do work for you by being really bad at it. And I don't love that pattern, but I'm still reminded of that because there are so many different ways to achieve any outcome that, okay, I want to run a ridiculous hot dog or not hot dog style website out there. Great. I can upload things. Well, Docker or serverless, what provider do I want to put it on? And oh, by the way, a lot of those decisions very early on are one way doors that you don't realize are crossing through as well as not well, Docker or serverless? What provider do I want to put it on? And oh, by the way, a lot of those decisions very early on are one-way doors that you don't realize are crossing through

as well as not knowing what the nuances of all of those things are. And that's dangerous. I think people are also learning the vendor as well. Some people get really engrossed in, whether it's Amazon or Google or Hashtag Corp or somebody's API, and you spend so much of your brain cells just learning how these people's systems work

versus general programming practices or whatever. I make it a point to build something on other cloud providers that aren't Amazon every now and then just because I don't want to wind up effectively embracing a monoculture. Yeah, for sure. I mean, and I think that's kind of the trend I see with people looking just at the Kubernetes stuff or whatever. It's that I don't think it necessarily exists in web dev. There seems to be a lot of, still a lot of creativity and different frameworks there, but people are kind of, what's popular, what gets me my next job and I kind of stumbled into building admin tools. I kind of made hammers, not houses or whatever. But basically, everybody was kind of building their own tools and deciding what they wanted. Now, like people that are wanting to make money are deciding what people want for them. And it's kind of not always the simplest, easiest thing.

So many open source projects now are, for example, one that I was dealing with recently was the AWS CLI. Great. I'm thrilled to throw in issues and challenges here, but I'm not going to spend significant time writing code against it because one, it's basically impossible to get these things accepted when all the maintainers work on Amazon. And two, is it really an open source project in the way that you and I think about community and the rest when it's basically sole purpose is to funnel money to Amazon faster? Like that isn't really a community

ethos I feel comfortable getting behind, to be perfectly honest. They're a big company. They can afford to pay people to build these things out full time. Yeah. And GitHub, I mean, we all mostly, I think, appreciate the fact that we can host a Git repo and it's performant and everything, and we don't have blazing unicorns quite as often or whatever than it used to have. But it kind of changed the whole open source culture because we used to talk about things on mailing lists like what should this be and there was like an upfront conversation or it might happen on IRC and now people are used to just saying I've got a problem fix it for me or they're throwing code over the wall and it might not be the code or feature that you wanted

because they're not really part of your thing. So before people would get really engrossed with just a couple projects. And if they were working on them as kind of like a collective of people working against different organizations, we talk about things and they kind of know what was going on. And now it's very easy to get a patch that you don't want. And you're like, oh, can you change all these things? And then somebody's like, now they're offended because now they have to do all this extra work. Whereas that conversation didn't happen.

And GitHub could absolutely remodel themselves to encourage those kinds of conversations in communities. But part of the death of open source and the fact that now it's give me free code is because of that kind of absence of the things. We're looking at that as like the front of a community versus like a conversation. I really want to appreciate you taking so much time out of your day to basically reminisce about some of these things, but on a forward-looking basis,

if people want to learn more about how you see things, where's the best place to find you? Yeah, so if you're interested in my blog, it's pretty random, but it's michaeldehaan.substack.com. I run a small emerging consultancy thing off of michaeldehaan.net. And that's basically it. My Twitter's laser llama if you want to follow that. Yeah. Thank you very much for having me. Great conversation. Definitely making all this technology feel old and busted, but maybe there's still some merit in going back. Old and busted because it wasn there's still some merit in going back.

Old and busted because it wasn't built this year? Great. Yeah, it's legacy, which is a condescending engineering term for it makes money. Yeah, there's an entire universe of stuff out there. There are companies that are still toying with virtualization. Is this something we get on board with? There's nothing inherently wrong with that. I find that judging what a bunch of startups are doing, or a company that started today, is a poor frame of reference to look at what you should do with your 200-year-old insurance company. Yeah, like whole software engineering is just ridiculously new. Like if you compare it to like bridge building or even electrical engineering, right?

The industry doesn't know what it's doing and it's kind of stumbling around on trying to escape local maxima and things like that. I will, of course, put links to where to find you into the show notes. Thanks again for being so generous with your time. It's appreciated. Yeah, thank you very much. Michael DeHaan, founder of Cobbler, Ansible, and oh, so much more than that. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud.

If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice and or smash the like and subscribe buttons on the YouTubes. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, smash the buttons as mentioned, and leave a loud, angry comment explaining what you hated about it that I will then summarily reject because it wasn't properly formatted yaml. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS, we tailor recommendations to your business and we get to the point.

Visit duckbillgroup.com to get started. This has been a HumblePod production. Stay humble. this has been a humble pod production stay humble