Command Line Heroes - Talking to Machines: LISP and the Origins of A.I.

Episode Date: September 17, 2019

Creating a machine that thinks may have seemed like science fiction in the 1950s. But John McCarthy decided to make it a reality. And he started with a language he called LISP. Colin Garvey describes how McCarthy created the first language for AI. Sam Williams covers how early interest in thinking machines spread from academia to the business world, and how—after certain projects didn’t deliver on their promises—a long AI winter eventually set in. Ulrich Drepper explains that the dreams of AI went beyond what the hardware could deliver at the time.But hardware gets more powerful each and every day. Chris Nicholson points out that today’s machines have enough processing power to handle the resource requirements of AI—so much so that we’re in the middle of a revolutionary resurgence in AI research and development. Finally, Rachel Thomas identifies the languages of AI beyond LISP—evidence of the different kinds of tasks AI is now being prepared to do.If you want to dive deeper into LISP and the origins of artificial intelligence, you can check out all our bonus material over at redhat.com/commandlineheroes. You’ll find extra content for every episode.Follow along with the episode transcript.

Kasparov is rattled. He defends what he can, and he recognizes that he is already lost. In the spring of 1997, world chess champion Garry Kasparov lost a match to the computer program Deep Blue. It was a pivotal moment in machine intelligence. For some, it was an existential

crisis, a challenge to the supremacy of human intellect. But for the technologists of the world, this was a milestone of another kind, a leap forward in the arena of artificial intelligence, a sign that their dream of a truly intelligent machine might not be so crazy after all. A machine that can think remains the dream. And it's still many years and quite a few startling breakthroughs away. How did we get to that point, though? What breakthroughs led up to Kasparov's famous defeat? And where did we go from there?

I'm Saran Yitbarek, and this is Command Line Heroes, an original podcast from Red Hat. All season long, we're exploring the mysteries of programming languages, uncovering their history and their potential. This episode, we zoom in on artificial intelligence. What language do you use when your tech has a mind of its own? How did our programming languages help us get to that deep blue moment and way beyond? The question of what language can work best with a thinking machine is something we've been tackling for more than half a century. And so, our story begins way back in the 1940s

when the term AI hadn't even been coined. Think back to the end of World War II. There's this sense that technology was how the Allies won the war. There was this optimism that technology could accomplish anything. A whole generation believed in the power of computing.

And out of that generation comes the godfather of artificial intelligence, John McCarthy. He's this brilliant mathematician who radically shook up the way we talk to our machines. I wanted to know about the origins of his ideas and how that paradigm shift came about. I mean, for starters, when McCarthy and his peers were imagining the future of intelligent machines, what exactly did they imagine? Wow, that's a good question.

I got talking with Colin Garvey. He's a historian in the Science and Technology Studies Department at Rensselaer Polytechnic Institute. Here's some of our chat. So McCarthy is remarkably kind of reticent about proposing exactly what it's going to look like. But for example, he has probably his most famous program that was actually never implemented, but it's kind of a thought piece, was The Advice Taker. And so The Advice Taker was written up in 1960 by McCarthy

in a paper called Programs with Common Sense. And it starts off, you think the advice taker is probably a robot that will learn, and that's his intention, right? Kind of a domestic robot. It could take advice. You say, no, you're doing that wrong. You know, do it this way, and it'll understand what you mean. That's helpful. Could be helpful. The whole goal of the advice taker is to drive from his desk to the airport. Basically, formalized his notion of what the advice taker robot should do, which is apparently drive him to the airport. He formalized it into a series of logical statements that it would need to infer about what needs to happen in the current situation to turn the current situation into the desired situation.

So he formalizes this in kind of a pseudocode, we'd call it now. And so this is actually where Lisp originates as well. And then in the next few years, Lisp comes out as his implementation or his language for implementing it. McCarthy's Lisp language was a game changer. It would help a machine understand not only commands, but a common sense logic. What McCarthy discovered was that he could write conditional expressions, code that embodied rules rather than just blunt commands. In fact, there were a bunch of major programming breakthroughs with Lisp. Conditionals, garbage collection, recursion, and lots more. Lisp used lists for both code and data, a simple change with profound effects. All this opened a gateway

for an entire field that McCarthy himself dubbed artificial intelligence. It was a language paradigm shift. Imagine speaking to a machine without giving it every particular of information. Imagine inviting that machine to infer and reason. Through his Lisp language, McCarthy hoped to give a kind of intelligence to the machine. Okay, back to my chat with Colin Garvey. It refined earlier attempts at writing a high-level computer language. Interesting, because that was going to be, you know, my next question was about the relationship between Lisp and AI. So it's almost like that first idea of what AI could do, that advice machine

was, you know, the beginning, it sounds like, of Lisp. Tell me more about that relationship between Lisp and AI. Sure. So one of the things that these early AI folks were up against was that they're doing programming with punch cards. And probably these early guys knew how to program at the level of machine code. And that's very time consuming and difficult. And so you needed higher-level languages that you could instruct in ways that were closer informed to human language. So something like Lisp gives instructions in literally a list of, and that's where the name comes from, list-based processing, in a list of instructions that are much closer to, I mean, the human language in the sense that they are basically logical propositions. So if you could read formal logic, you can basically look at a Lisp program

or any of the logic-based programming languages and have a much better sense of what's happening in the code. So Lisp really helped. Yeah, I mean, it helped us take our ideas of artificial intelligence and actually work towards them and make them happen, which makes me wonder what intelligence even means during that time period. So if we go back to the 50s at that point, what was intelligence? How did people even define that back then? Because LISP was first developed for the IBM 704, which does really just one thing at a time. So it doesn't really sound very intelligent. Yeah. How did people think about intelligence at that time? So this is, of course, incredibly controversial, right?

They have very narrow concepts as far as I'm concerned from a social perspective. But yeah, at the time, for instance, the ability to perform a behavior that would be described as intelligent is kind of the catch-all definition. But these guys are really mathematicians and logicians and computer programmers. I mean, to be crass, the ability to play chess was considered a definite sign of intelligence. And this early generation of AI people was much more willing to sidestep this question and say, well, the philosophers have not agreed on what intelligence is, but if we make a computer that can play chess, I think we can all agree that's intelligent. It was a starting place anyway. Baby steps. McCarthy had a dream that machines could be intelligent as people and have common sense. And essentially, you could talk to them.

And he set about creating a program language to make that dream a reality that became Lisp, and it captured certain aspects of human thought, and especially logical thought, and made it possible to use computers to amplify or extend those features of thought. So from a mathematician's perspective, he was well on his way to realizing intelligent machinery. Colin Garvey is a historian in the Science and Technology Studies Department at Rensselaer Polytechnic Institute. And so with LISP, new opportunities to advance AI began to open up. And Lisp was the standard language in that new frontier. After McCarthy left his post at MIT for a job at Stanford, other engineers back at MIT kept working on Lisp.

They refined McCarthy's language into a local dialect called MacLisp and even developed a Lisp operating system. It looked like McCarthy's dream of an AI future was well on its way to becoming a reality. In 1968, McCarthy even made a bet with David Levy, a Scottish chess master. McCarthy bet him that in 10 years, a computer would be able to win a chess match against Levy. But it wouldn't be a command line hero's kind of story if things went that smoothly. McCarthy didn't know it yet, but the AI winter was coming. Things started to change when that Lisp-based operating system from MIT was spun into companies. One of those companies, Symbolics, introduced Lisp machines and even hired away 14 staffers from MIT's AI lab. And that earlier, more blue-sky kind of research

wasn't the focus anymore. You get all the spin-off companies that were coming out of the AI lab. This is journalist Sam Williams, who wrote a book called Arguing AI. Symbolics is probably the one that sticks out the most, and it was getting the most attention, the most venture funding.

John McCarthy's, all his protégés are kind of driving all that innovation. I think it's just a case, I think, where, you know, at the end of the 70s, there was a ripe period where people were like, okay, well, we've done what we can inside academia. Let's get this out in the business world and let the private sector finance this. Which made sense at the time. AI looked like it was about to take off. There was money to be made. The computing power is cheap enough for a mid-level business to purchase. So I think a lot of companies saw an opportunity. Okay, we have a market for this. We can get out there and we can sell this to businesses that are trying to be globally

competitive. And they got investors to buy that pitch. There was a rush of money into the field, you know, in the early 80s. We can argue about why that rush of money ran dry later down the line, but here's what we know for sure. Things had been incredibly hyped. A lot of money flowed in, you know, People saw the market as very ripe. In a way, that cash boom in AI was like a prelude to the dot-com bubble that hit at the turn of the millennium. As happens a lot in technology, people overinvested and eventually companies failed to deliver on what they were promising. In the course of that, then there's a sort of shut off to the money supply and companies have to make their money the old fashioned way by actually building customer bases and stuff.

So now McCarthy's dream was crashing back down to reality. Williams describes the AI winter that followed. You see a company like Symbolics, you know, rises up to $100 million market cap and then a few short years later is filing for Chapter 11. There's a lot of, you know, in 10 years we'll do this, in five years you'll be seeing this. These guys are hype merchants, or they're just academics who are trying to, you know, squeeze more funding out of the Pentagon or wherever they're getting money. And so, you know, there's this dangerous hype, overhype, AI, let's move on to something else, phenomenon.

Interest in AI just crashes. It didn't help that the infrastructure wasn't there or that computing speed wasn't high enough. But at any rate, this was a case of right idea and wrong decade. By the late 80s, AI research was dwindling. I think that people got away from what drew them to AI as a field, which was very adventurous. It was a chance to do the cutting edge. Out on the West Coast, a lot of the energy is getting drawn into, you know, let's make personal computers. I think from 40 years later, the attempt to get a computer into every home in the United States is a way more compelling concept than, you know, helping some Fortune 1000 business make better decisions.

Sam Williams is the author of Arguing AI. What we're describing here is a world of computer programming that was split into two. On the one hand, you've got the John McCarthys trying to use new kinds of languages, like Lisp, to inquire into the nature of intelligence. On the other hand, inevitably, you've got folks trying to solve practical problems and make things easier for businesses that are willing to pay for services. The reality was that the John McCarthys of the world, those more abstract dreamers, didn't have much of a chance to explore deep

in the possibilities of AI. The equipment they had wasn't even comparable to the cell phones we use today. Their vision was really too big for the hardware realities on hand. We had machines, big honking machines, which are used by many, many people in the work groups. That's Ulrich Dreppe, a distinguished engineer at Red Hat. He explains how that early AI dream just didn't have the basic tools it needed to succeed. So we have PDP-10s, PDP-11s, etc., these kind of machines, where you might have 20 people on them.

At best, they were really low power machines, which didn't really have the capability of running a nicely developed user interface in addition to all the functionality necessary for a LISP system. As decades rolled on, the hardware that was being developed wasn't friendly toward Lisp either. We went down the road for the desktop machines, but also for the server machines, of the type of processors which we are accustomed to today. So we're having divisions inside the processors based on 8-bit bytes. So we might have a 16-bit processor, 32-bit processor, etc. So for these types of machines to implement in this system efficiently, one had to jump through quite a few hoops to actually make it work. So external realities about the computing world were having unintended dampening effects on this fragile field of artificial intelligence. The custom hardware turned out to be such an expensive and slow prospect, especially when you compare it to the CPUs which were in common use, the big CPU manufacturers.

It was possible to create those custom hardware workarounds and keep working with Lisp, keep trudging toward that AI dream. But the fact was, exciting new hardware pushed a lot of minds toward other areas of work. The improvements you got by just waiting for the next revision of the processor, the normal processor was so much bigger than what you can achieve and gain by developing your own hardware for solving, for instance, for implementing a list system. So that in the end, the complexity simply didn't work out. So, was John McCarthy's wide-eyed dream dying a slow death? Or was it only sleeping through that long AI winter, waiting for its moment?

Through the 2000s, John McCarthy's language, Lisp, was used less and less. But something kind of amazing happened at the same time. The language began to fade, but the dream connected to that language got rekindled. The dream of AI began to live again. I want to say this time is different. That's Chris Nicholson, the founder and CEO of an open-source AI software company called Skymind.

Chris figures that after all its ups and downs, AI may be coming into a long, sustainable summer at last. We're in the midst of an AI summer where there's a lot of excitement around what AI can do. But that excitement is based on real advances in research. And those advances are based on progress in other areas. Our hardware has gotten a lot better. People are able to use GPUs instead of CPUs, and they're able to use those GPUs in massive clusters to train enormous machine learning models that can produce really accurate predictions. They do that using massive data sets, more data than we've ever had before,

much more data than we had in the 90s or 80s or before. When you're thinking about AI languages, the rise of enormous data sets is key because they represent a crucial change in the way we understand what intelligence even is. The focus shifts and becomes all about deep learning. Lisp was designed to manipulate symbols, right? Lisp was designed in an era when AI meant symbolic reasoning, when people thought that the human mind and intelligence itself was simply a manipulation of symbols. And AI now, the AI we're really excited about, is much more than that. AI now and machine learning now is a massive data processing machine that can learn from unstructured data. So blobs of pixels, blobs of text, right? You can shove them through machine learning algorithms and make those algorithms smarter and smarter over time as you push the data through.

Lisp wasn't designed like that. It was designed to only evolve on the back of human intervention, whereas machine learning algorithms now, the learning part of them means that they adapt themselves on the basis of the data they're exposed to. This is about letting our machines be intelligent on their own terms. But you know, it's not just our machines that are learning to think in powerful new ways. Open source tools have allowed people to up their game too. And Chris figures our open source development styles are going a long way to preventing another AI winter. One reason why winters happen is because ideas slow down, networks break down, right?

Funding dries up. And with these free open source tools, what we see is an acceleration. So we see ideas getting tested faster, ideas getting shared more quickly. So it's not just the tools, but it's pre-trained machine learning models that one research group might share with another research group or with the public at large. And finally, it's the ideas themselves, which are getting published in locations like Arquive, right? AR XIV, which is hosted by Cornell University. So all those things together are accelerating kind of the velocity of the flow of ideas. And for me, that's kind of insurance against an AI winter.

There's this confluence of factors that makes AI a super exciting place to be. And that actually means AI is moving even faster than the hype. Chris Nicholson is the founder and CEO of SkyMind. So, open source was able to thaw out the AI winter that older proprietary models had helped to create. The new AI moment came with vast improvements in GPUs. It came with algorithmic breakthroughs that help us train AIs to learn patterns in data.

And then the data itself. Massive data sets are fueling today's AI. And that, more than anything, is why new languages are picking up the trail where Lisp left off. We asked Rachel Thomas, a co-founder of Fast AI, about which languages are suited to next-gen AI. We're starting to look at Swift for TensorFlow. So Swift is a language developed by Chris Lattner. And I know there are people using Julia and developing Julia. But for right now, Python is kind of overwhelmingly what is used in deep learning.

That's because Python is beautifully suited to working with large data sets. So I think something that's really changed in the last 20 years is the rise of data and the increase of data. And I think that a lot of earlier work on AI going back decades was more around kind of symbolic systems. Yeah, these kind of abstract symbolic systems. And I would say a lot of the work happening in AI right now is very much around

data and, you know, identifying something in an image, classifying whether a view somebody's written of a movie is positive or negative. Kind of these are problems that are grounded in data. And so one of the reasons I think Python has kind of emerged, at least as an early winner, is because the ecosystem with NumPy and SciPy and Pandas and all the data libraries in Python were well-developed enough. In the days when John McCarthy's group worked at MIT, AI really was something only the world's elite academics could get involved in. But languages and computing environments have evolved so much since then that AI is now, really, for everybody. Like we heard in episode one this

season, Python is a first language for lots of beginners, and it's empowered novice programmers to join the world of AI. I just want to tell people that it really is just one year of coding experience is the only background you need. You don't need to be, quote, a bath genius or to have this really prestigious or exclusive background, but that we need people from all backgrounds. And there's by actually by having a different background, you have a lot to offer to the field and that we need your perspective. Rachel Thomas is a co-founder of Fast AI and a professor at the University of San Francisco Data Institute. I love that message from Rachel Thomas. The story of AI is a story about discovering languages that everybody can communicate with. That's our real mission here. To develop language solutions for a world filled with AI.

Solutions that let us all work together to build that next breakthrough. Folks, you saw history made here today. AlphaGo has won the match in great style. Yeah, and it even showed us its code. And even showed us its code at the end. So I think congratulations to the program. In 2015, just a couple decades after Garry Kasparov lost that chess match to Deep Blue, the world champion at the game of Go, Lee Sedol, was defeated by Google's AlphaGo.

Deep Blue had shown its intellectual might, pretty much through brute computational force. The thing about AlphaGo's victory that really surprised everyone was that it won that match by relying on neural networks and reinforcement learning. In other words, AlphaGo had successfully taken that next step toward true intelligence. None of that would have been possible without the new AI spring, fueled by open source. New languages are edging toward that dream. Our languages are getting better at maximizing the intelligence of machines. The good news,

it's a challenge we can all work on together in an open source world. Command Line Heroes is an original podcast from Red Hat. If you want to dive deeper into Lisp and the origins of artificial intelligence, you can check out all our bonus material over at redhat.com slash command line heroes. You'll find extra content for every episode. Next time, it's our season finale. We'll uncover some surprising truths about a massive language

that influenced nearly every other language we studied this season. It's our in-depth look at C. In the meantime, I'm Saran Yitbarek. Keep on coding. Hi, I'm Mike Ferris, Chief Strategy Officer at Red Hat.

And as you might expect in my role, I get a lot of questions about AI, particularly about foundation models. Thank you. And you need a common platform for that where data scientists, app developers, and ops teams can all collaborate, especially as you start to scale. And then this is iterative. It's rinse and repeat. So really, it's about making that fast path from idea to model to production and back again. And that's what Red Hat OpenShift AI does. Head to redhat.com to learn more.