Command Line Heroes - Heroes in a Bash Shell

Episode Date: September 3, 2019

Shells make large-scale IT possible. They’re a necessary component to modern computing. But it might not have turned out that way without a lot of hard work from a developer at the Free Software Foundation named Brian Fox. Now, the Bash shell is shipped with almost every computer in the world. In the ‘70s, Bell Labs wanted to automate sequences of repetitive, complex commands. Chet Ramey describes how Bell developed several shells—but there could be only one officially supported shell for UNIX. Enter the Bourne shell. Though it was the best of that crop, the Bourne shell had its limits. And it was only available with a limited UNIX license. Brian J. Fox recounts his time at the Free Software Foundation where he needed to create a free—as in speech—version of the Bourne shell. It had to be compatible without using any elements of the original source code. That Bourne-Again Shell, aka Bash, is possibly the most widely used software in the planet. And Taz Brown describes how it’s one of the most important tools a developer can learn to use. You can dive deeper into the story of Bash, or any of the programming languages we cover this season, if you head over to the show’s site at redhat.com/commandlineheroes Follow along with the episode transcript.

It's 1987. Ronald Reagan's America is in full swing, and a 27-year-old high school dropout with big dreams is driving to his new home in Santa Barbara. That man's name is Brian Fox, and in the trunk of his car, he's got two massive tapes filled with the code he's been writing. Fox has spent years working as a programmer in something called the Free Software Movement. He believes the code he's got locked in that trunk is part of a new reality, a new software paradigm that his community is bringing to life, one piece at a time. That year, a team of coders at Richard Stallman's Free Software Foundation were trying to set the programming world free. They wanted to build an alternative to the Unix operating system that had dominated programming since the 70s.

Their GNU, which stood for GNU's Not Unix, was going to be an operating system for the people, one that anybody could use without worrying about license fees or copyright. The foundation had been cobbling this brave new operating system together for years. And those two massive tapes of code in the trunk of Brian Fox's car? They held a crucial component. Written on those tapes was a free and hackable shell that could complete the GNU operating system. Brian Fox's gift to the free software movement. He called it Bash. I'm Saranya Dbarik, and this is Command Line Heroes, an original podcast from Red Hat. This episode, we're looking at our heroes in a Bash shell. We're uncovering the history of shells and why they're so crucial to our work today.

Think of shell scripts like a script you would give an actor. They deliver a whole sequence of commands that the shell can then rattle off on its own, the same way an actor can read her lines one after another. It's the ultimate workaround if you've got repetitive or convoluted commands. It's the key to automation. You might say that shell scripting supercharges our development. But could a shell be written that gave that superpower away to the whole world? That was the challenge. Back in 1969, a couple of computer scientists here at Bell Labs

started to develop some programs they needed for their own use. That's pioneering command line hero, Ken Thompson. The Unix operating system designed at Bell Labs really was intended for their personal use. Originally, it was just an internal system. Unix encouraged close communication among programmers, but it wasn't intended to change the way the whole world worked. It was intended to change Bell Labs. By now, it's used all over Bell Labs. We have close to 20,000 computer terminals in this company, and most of them are used for communicating with Unix systems. A Unix shell designed by Ken Thompson was released in 1971.

The Thompson shell was designed to be a command line interpreter, but it really wasn't capable of full-on scripting. It wasn't until six years later, in 1977, that scripting started to take off. The shell parameters, the special parameters, the variables that we sort of take for granted today originated with Steve Bourne and the Bourne shell. That's Chet Ramey, an IT architect at Case Western Reserve University. Chet works at maintaining Bash, but he's also a great resource for our origin story. He describes Bell Labs, just as it was figuring out what the Unix shell

was going to look like. The programming constructs that we use without thinking today originated with Steve Bourne, and his shell basically won the bake-off. There was a significant user community using the Mashi shell. There was a significant user community beginning to use the Bourne shell. There was a committee that was set up to decide which one would win, which one would be the officially supported Unix shell coming out of Bell Labs from then on, and Born Shell won. And the rest, as they say, is history. It's not the end of history, though. Sure, the Born Shell was a huge leap forward. It opened a door towards super-powered operations, toward greater automation.

Yet, while there was a kind of Bourne supremacy for a while, the Bourne shell didn't solve all our scripting needs. The constraints under which Bourne wrote his shell are almost unimaginable today. Obviously, when you have those constraints, you have to give up a lot. And Born gave up a lot. It's remarkable that he was able to put as much into the Born shell as he did, given the space memory and CPU constraints he worked with. And remember, the Born shell was still part of Bell Labs' Unix system. It was still tied to the Unix license. That meant it wasn't free. It wasn't open. This shell was owned. It was incredibly difficult to get Unix source if you were not at a university. And obviously that had an effect on the popularity of Berkeley Unix, for instance, which started at a university, grew up among a community of universities,

and kind of took off on the path of least resistance, as it were. So getting access to the Born Shell code was not difficult if you were at the right place. But in general, it wasn't viable. Chet Ramey is the maintainer of the Bash Shell. So we've got the beginnings of shells, the start of this crucial component to programming. But the best shell out there is tied to a license. It's closed. For Richard Stallman and his Free Software Foundation, that arrangement just wasn't okay. What was needed was a shell that wasn't tied to any one company.

A shell for the people. But here's the trick. That meant writing something that did everything the Bourne shell could do without infringing on any of those pesky copyrights. Copy the Bourne shell's code verbatim and you'd have a lawsuit on your hands. To free people from the Bourne shell, you'd have to find a coder with the ability to write a program that complex, a program that did everything the Bourne shell could do, but who hadn't actually seen any of the Bourne source code. You'd have to find a kind of outsider genius. And Richard Stallman had just the coder for the job. Brian Fox was a 20-something high school dropout who knew code better than most of the folks at Bell Labs. He'd never been in a position to see any of the source code that made the born shell work, and that made him ideal for the task at hand.

My name is Brian Fox. I figured, why not get this story from the man himself? These days, Fox is an open source advocate and the CEO of Opus Logica. But back in the late 80s, he was just a young guy who believed in the free software movement. We chatted about the old days and how Bash evolved from there.

That is a good point. Okay. So Richard Stallman asks you to create a shell for Unix. That one will be a free, free shell. And it's a replacement for the Bourne shell. What was your response to that request? Can't we make a better one? Yeah.

I love that. Tell me more. So the very first thing I did for Stallman was actually work on this tech info documentation system. I surprised Richard at the speed at which this type of programming would be done. He's a good programmer and he works quickly, but he doesn't judge that other people would work that quickly. So within the first week, I finished the first implementation of a program called GNU Info. And Richard was kind of stunned by that. And I said, what's my next project? What's my next project? And he said, well, now do a compiler for this. And then I did that and I was done in a week with that. And then I said, what's my next project? What's my next project?

And he said, well, this other guy has been working on the shell, but he hasn't gotten very far. I was like, okay. And nine months later, the Bourne replacement shell was done. Nine months. Wow. Tell me about that. Why was it so challenging? You know, that's actually a fascinating question.

The reason it was so challenging is because we had to faithfully mimic all of the behaviors of the Bourne shell, of Stephen Bourne's original shell, while at the same time being allowed to extend it and make it a better tool for people to use. And while that was happening, I was in a quiet undercover argument with David Korn, the author of the Korn shell, the POSIX committee, which is the committee that says what's standard Unix, they got involved and said, oh, good, we're going to define what the standard shell is. And the two most important people on input for that were myself and David Korn. And David Korn had already written this shell called KSH. And every feature that he had put into KSH, he said that should be a standard feature, right? This would be easy for him then to have the most perfect POSIX shell if it was simply his shell. And some of those features were not good features, were not good choices and made the shell somewhat incompatible with

the born shell or I felt were misfeatures. And so there were several discussions and arguments about that. And so building a POSIX compliant shell that was 100% perfectly compatible with every single shell script that had ever been written for the Bourne shell took longer than three months. So if you are designing something that not only replaces the Bourne shell, but also is trying to mimic every part of the Bourne shell, it sounds like you might have run into some copyright issues. How did you approach that? In order to build true open source and free software,

you have to do it in a clean room. You can't look at somebody else's code, start from there, and reimplement it. So I'd never seen any of the software associated with any Bell Systems Unix or even Berkeley Unix. I'd never seen the source code for any of these things. When I started building the Bash shell, I used a parser called Bison, which was something that Richard had started to put together at the Free Software Foundation. And that was completely different from basically any other program that had come before it.

So I knew already that the thing I was building was not ever going to be a copyright infringement on something that had been built previously. The work to create Bash had plenty of hiccups. Here's just one example for the hardcore heroes out there. At one point, I was working on implementing globbing in the shell. This is the wildcard expansion that allows you to match a large number of files, for example. You could say star.c, and that would match every file that had an extension of.c. So I worked on globbing for several hours, and I got it working, and I was excited about it. It was a good implementation. And in the course of creating this implementation,

I had created a file in my directory called asterisk.c, you know, a star.c. And I thought, well, I should get rid of that file. And I typed in rm, quote, star.c, close quote, which in a modern shell, when you use the quotes, it means do not expand this. And then I pressed pressed return and it was taking a long time for the prompt to come back because we're using sun 350s and things are slow and i realized it's taking a long time because it's deleting all of the source files in this

directory oh no yeah so i deleted the source to bash at that point. Oh no. Oh my goodness, yeah. Which caused me to just laugh kind of loudly for a really long time. I wasn't even slightly upset. And then over the next couple of days, I typed it all back in. The code was completely

fresh in my mind. The problems had been solved. It was just a matter of putting it down into files. Okay. So most people would completely freak out at that moment. You laughed and you just said, oh, I guess I have to do it all over again. Why were you so calm? It struck me as insanely absurd and very funny that I'm building this tool and to make sure, you know, it's good to eat your own dog food, to make sure the tool works correctly, you use the tool while you're building it. Yeah, yeah.

But the tool didn't work correctly. I had not yet implemented quoting. And because I hadn't implemented quoting, this command that I casually typed did not do what I expected it to do. Yeah. And I thought that was really funny. And then, I mean. That's amazing. Even that story about a mistake speaks to Fox's brilliance, though.

They say that Mozart finished symphonies in his head and then just had to write them down once he'd finished. Fox had a similar talent. So when you were finally done and you got to deliver Bash, how did that feel? It actually felt spectacular. So here's a story that I don't actually usually tell. It was about eight months into building the shell. I knew I needed about a month before I'd be done. And another shell was released. ASH, a open source shell, got released. And I was crestfallen because we had not released the Bash shell to anyone yet. So only a handful of people were using it.

I knew it needed another month's worth of work. And I thought, oh, this is terrible. All this energy and effort I've put in will not be appreciated. It might not even be seen. And so I was pretty distraught. I was not laughing. The proof was in the pudding, though. GNU's Bash was released in 1989 and became the default shell for Linux. Today, it saturates our whole computing experience. But it is everywhere. Someone who will use it every single day. It's on every single computer. How does it feel being the author of Bash? Most of the time, I don't even notice that Bash is a thing other than a tool that I use in my daily life. I don't really think about it. But every so often, I'll walk into an Apple store and look around and think, wow, every computer in here is not only running software

I wrote 27 years ago or more now, it also has my name in it. And then I think every computer on the internet, every server on the internet is running the Bash shell and has my name in it. And then Windows last year or the year before came out with the PowerShell, which is Bash. And I was like, oh my goodness, my name is in every computer on the planet. I want you to really hear what Fox told me next, though, because it's so important. He was never trying to code his way through this global domination. He was trying to help, trying to help the culture of programming that he was part of. I didn't set out to achieve some grandiose goal of being on everybody's computer.

I wasn't interested in that at all. I wanted to make a useful piece of software, and I expected it to have a kind of typical software lifespan of, you know, three to five years, not this kind of crazy 30-year, you know, term that it's had. Or were you always so, frankly, so nonchalant about the impact that you've had on computing. I'm proud that I wrote Bash, and I have an ego, so I do things like accept podcast requests to talk about the shell. Thank you very much. Thank you. But it is not something that is there in my everyday life. Fortunately, I'm just an obscure person, right? It is true that my software is running on everybody's computer in everybody's house.

And it's also true that nobody knows that. Yeah. Right. So I have plenty of anonymity. And the fact that the shell exists and that somebody wrote it and that person lives in Santa Barbara is getting more known. And I'm beginning to notice it more in my life. People sometimes come to see me play music and then tell me, you're the guy that wrote the shell. Yeah.

And I feel a little like Keanu Reeves. Very cool. So you said that you didn't set out to make sure Bash is on every single computer. What did you set out to do? What were your hopes for Bash? A useful replacement tool to be part of the Project GNU and to help create this free open source operating system. I actually assumed that once we had finished creating this open source operating system, the software on

that system would advance and I would get a chance to create the kind of shell that I wanted to create, which would help people to advance the science in a manner of speaking. I eventually came to the realization that the reason Bash was created was to, in fact, be backwards compatible with the entire world of Unix that already existed. And that momentum has kind of kept it alive, which is another unique position to be in. Yeah. That your tool is so fundamental. It's so much of a nut and bolt that it's not something that will be removed.

Absolutely. It's been a great feeling to know that I've created something of value in the world, something that other people are still using. And that is a good feeling. And then as I look at how that came about, I realized the more important thing is the words free software and open source are in everyday English, everyday language across the globe. And that certainly wasn't true on day one. That was a creation of the efforts that Richard Stallman and myself and others put in. And to be part of that movement, that's lucky to have been early, but it's also extremely satisfying when I look back at that and I think, wow,

open source software exists, and I was a part of that. Brian Fox is the creator of the Bash Shell and CEO of Opus Logica. I did hear about Bash, actually. That's Steve Bourne, creator of the Bourne shell that Brian Fox worked to replace. We wanted to know what Bourne thought about Fox's work. Did he think of Bash, that Bourne again shell, as an open source clone of his own work? I mean, was he cool with Bash?

The person that wrote it came out to me at a conference one day and gave me a T-shirt which said Born Again on the front of it. That'd be Brian Fox. It was a friendly sentiment and it was, well, I hope you don't mind, but I just rewrote your shell and I said, that sounds great. And he gave me a T-shirt. If there's one thing I've learned in the coding world, it's that everybody loves swag. Stephen Bourne, turns out, saw Bash as a necessary extension of the work he and others did at Bell Labs. There's no bitterness at all. There were things that people wanted to do in variable substitution and doing string

management that I didn't do, but they were put into Bash that people use a lot these days. The relationship between Bash and the original shell, my impression at the time was it was just a reimplementation of the language. And over time, it did have features added to it. So it did sort of progress beyond what I wrote, certainly in the string management area. I use it all the time now.

Steve Bourne is creator of the Bourne Shell and CTO at Rally Ventures. It's been many years now since Bash was stowed away in Brian Fox's trunk on that long drive to Santa Barbara. In 2019, version 5.0 was released. And, like Fox mentioned, Bash is now built into Linux, into macOS, and even into Microsoft Windows. As Unix gives way to Linux, Bash has become a cornerstone of scripting in an open source world. It's fundamental to our automation. It became almost crucial as organizations got bigger to use something that would allow us to get things done quicker. It became a necessity.

Taz Brown is a senior Ansible automations consultant at Red Hat, so she's well acquainted with the value of Bash. I definitely think that someone starting in the beginning stage of their career should definitely use Bash. Instead of using the GUI or the graphical user interface, you tend to be more of a, taken more seriously as an admin or as a DevOps person. And that's because a Bash coder will have one of those core skills that simply levels you up. There's a value in learning

scripting because it prepares you to be much more of a long-term sort of thinker when it comes to automation itself because you can see how a script runs. Then you can start to say, well, okay, I can do this. I can automate this task. I can automate this. And it starts to make you a different thinker and a different technologist. For the op side of things, that automation has become indispensable. Sophisticated programs, applications, and tools are all being supported by legacy bash code. You don't have to reinvent the wheel, if you will. You can continue and just pull those in from a GitHub repository or wherever you store those particular files. Bash allows you to do that.

Bash allows you to take those common tasks and allow you to scale across to, say, from 10 servers to 1, thousand servers. The great thing about automation is once you have a plan in place, it allows you to do it at a very cost sort of efficient manner. It allows you to do things that would be impossible to do manually. And then more recent arrivals like Ansible, which Taz Brown works on, can always be integrated with Bash to get the job done. Things have evolved, but I don't think Bash is ever going to not be a tool that an admin would apply, especially if you want a quick automation. In the end, all this success can be traced back to the fact that it's a free and hackable program. Brian Fox's desire to give something to the world with no licenses, no strings, has been the key to Bash's success. In fact, he's not even calling the shots anymore.

Hasn't for a long time. Here's Chet Ramey again, who's been maintaining Bash for decades. Brian had decided after releasing, I think, version 1.05 that he wanted to move on and work on other things. He had been given other assignments at the Free Software Foundation, and he wanted to do things besides Bash. And I was the most active contributor. He and I worked together on a lot of new features. We worked together on a lot of bug fixes. And so when it came time for someone else to take over, I was the logical candidate. And Ramey will have to pass on the mantle too, just like Fox, because Bash is bigger than any one maintainer. I started when I was 23, and Bash and I have kind of grown together. At some point, I will need to solicit a team. I will need to solicit folks who are willing and able to put the time in

and move the shell forward. Bash, the born-again shell, will turn 30 next year, and it's showing no signs of shrinking away. Bash has ridden the free software wave, and then the open-source wave, until it's spread to every corner of the programming world. But it's amazing to remember that, at one point, it was just code on a tape in the trunk of Brian Fox's car. It was just a dream of a shell language that a few

committed coders were willing to give away. Almost by accident, Brian Fox became a huge command line hero in the process. By the way, something's been bugging me. Brian Fox driving all that bash code to Santa Barbara. Why the move? I mean, did he have a new job at some tech company or? I wanted to continue my music career. And I thought the best place to do that was where the weather is always about 72 degrees and there are no clouds in the sky and the beaches are beautiful. Nice. I like that reason better. Shout out to Wayne A. Lee, who suggested our title for this episode, Heroes in a Bash Show. Nice one, Wayne.

Next episode. We take our interest in automation to a whole new level and look at the languages of AI with a special focus on John McCarthy's creation, Lisp. Command Line Heroes is an original podcast from Red Hat. You can dive deeper into the story of Bash or any of the programming languages we covered this season if you head over to the show's site at redhat.com slash command line heroes. I'm Saran Yadbarek. Until next time, keep on coding. Hi, I'm Mike Ferris, Chief Strategy Officer at Red Hat. And as you might expect in my role, I get a lot of questions about AI, particularly about foundation models. Now, don't get me wrong.

Those are important, but they're not the whole story. Whether you're using a commercial model or an open source one, you're going to need to fine tune or augment models with your data for your use case. And you need a common platform for that

where data scientists, app developers, and ops teams can all collaborate, especially as you start to scale. And then this is iterative. It's rinse and repeat. So really, it's about making that fast path from idea to model to production and back again. And that's what Red Hat OpenShift AI does.

Head to redhat.com to learn more.