Command Line Heroes - Invisible Intruders

Episode Date: May 17, 2022

What began as a supposed accounting error landed Cliff Stoll in the midst of database intrusions, government organizations, and the beginnings of a newer threat—cyber-espionage. This led the eclectic astronomer-cum-systems administrator to create what we know today as intrusion detection. And it all began at a time when people didn’t understand the importance of cybersecurity. This is a story that many in the infosec community have already heard, but the lessons from Stoll’s journey are still relevant. Katie Hafner gives us the background on this unbelievable story. Richard Bejtlich outlines the “honey pot” that finally cracked open the international case. And Don Cavender discusses the impact of Stoll’s work, and how it has inspired generations of security professionals.If you want to read up on some of our research on ransomware, you can check out all our bonus material over at redhat.com/commandlineheroes. Follow along with the episode transcript.  

I know, I know. Yeah, I've got it with me. Okay, just ducking into the subway. See you soon. Oh, great. Twenty minutes before the next one. Carol waits on the freezing subway station, alone. Out of habit, she scans her phone. No signal. Looks in her bag. There's the thumb drive. After months of back and forth with a reluctant source, Carol finally has the incriminating evidence she needs to win her case. What's in her bag proves it all.

Carol looks to the opening elevator door. No one comes out. Strange. She takes a few steps backwards and looks down the rest of the platform. Anybody there? Hello? Hello? Is somebody there? Clutching her bag, Carol starts to make her way to the stairs. She can hear someone. She just can't see them. What do you want? An invisible

force grabs her bag, wrestles it from her grip. The thumb drive, with all those crucial secrets, disappears. The ability to turn invisible, to roam without being noticed, has profound consequences. It opens the door for all kinds of crimes. After all, the rules change when nobody knows you're there. Suddenly, the risk factor drops away. Forbidden places, forbidden actions. They all start to look inviting.

I'm Saranya Tbarek, and this is Command Line Heroes, an original podcast from Red Hat. This season, we're exploring the great security crises of the digital age, and we saved one of the wildest stories until now, the tale of a hacker who crept unnoticed into some of America's most valuable computer systems. They snooped wherever they wanted wanted because they believed nobody would ever see their movements. Nobody would ever track them. It was a perfect plan unless someone found a way to begin seeing the invisible. It's the 1980s. The internet as we know it really doesn't exist.

There's ARPANET, and it was in some ways fantastic, in other ways really innocent. It was like a neighborhood where people feel really safe just keeping their doors unlocked. Katie Hafner is a journalist and author who's been writing about technology and hackers since the 80s. Forty years ago, she reminds us, computer networks were just beginning to grow, and security was not top of mind. Logins and passwords were a formality. The ARPANET was being built, after all, to help people connect. And unwanted intruders were... Back then, really kind of a stunning even notion

that someone would want to break into a computer. In the midst of that innocent ARPANET age in 1986, an excitable astronomer named Cliff Stoll was looking for work. While waiting for an astronomy job, he found himself a gig as a system administrator at the Lawrence Berkeley Lab in California. He'd helped them run a dozen mainframes that scientists logged in to use. Stoll chose a desk in an unventilated, windowless office in the basement,

hoping nobody would notice him. There he was, and he'd been asked to look into this small accounting discrepancy. Hundreds of scientists paid to use those mainframes, and the accounting logs at Berkeley Lab showed that for the first time ever, something didn't add up. Stoll's boss had found that 75 cents were missing. An ordinary person might have assumed it was some kind of rounding error and moved on. Stoll, though, wasn't ordinary. 75 cents couldn't just go missing. He started scrolling through the list of users and hours later found one that didn't have a valid billing address. He deleted that user from the system. Now, this is where most people would go for lunch, forget about the whole thing.

But this discrepancy didn't sit well with Stoll. He decided to watch the system more closely. He programmed his terminal to beep whenever somebody logged on to one of their computers. Every few minutes, Stoll heard a beep and ran over to see what username was being typed in. He asked his boss about one name that kept coming up, Sventec, and his boss found that odd because the user behind that name, Joe Sventec, was away that year. Sventec hadn't even been a user at LBL for a while. So, if Sventec wasn't around anymore, why was Sventec logging in? And then he realized that it was somebody who had basically taken over Sventec's account.

Who would want to do that? The Berkeley Lab's files were mostly dry academic stuff, and they weren't exactly top secret either. It seemed pointless to sneak in using somebody else's username and look at them. Stoll couldn't let it go. What happened next turned into a legendary cat and mouse story. Stoll even turned it all into a thriller-style memoir called The Cuckoo's Egg. You might have heard about it.

His book sprawls across continents and through a lot of technical details. But we're focusing on just one question. How do you catch an invisible intruder? Here's how he started. When he noticed there was an intruder, he took on the job again, just as a one-man SWAT team. Stoll slept beside printers that spat out records of every keystroke by each user that came through the lab's modems. He studied it all, and then he noticed somebody had spent four hours looking around their operating system throughout the night.

This was getting strange. And then it got even stranger. Whoever this intruder was, they had somehow learned to cover their tracks, to make themselves invisible. They could do this because they'd managed to make themselves into a super user. This person had keys to every room in the system. They could read everything. They could change everything. They could even delete the accounting files, which was how they were able to move around unnoticed. That 75-cent accounting discrepancy was the only mistake they made.

It was like a single fingerprint they forgot to wipe clean. Other than that one mistake, they had somehow figured out how to make themselves into an invisible intruder. Getting into the lab system in the first place would have been a simple version of what we'd call a brute force attack. The digital equivalent of trying car doors until you find one that's not locked. Back then, it was very common for a login to be somebody's name and then the password to be their last name or their birthday. It was all very loosey-goosey.

But once they were in, how did they get all those super-user privileges? For what happens next, you need to know that the Lawrence Berkeley Lab was using a variation of Unix called GNU. The way that Lawrence Berkeley Lab had set it up, they had installed a text editing program, and clearly the intruder knew some really basic Unix commands. The GNU Emacs editing program had its own email subsystem that had this tiny security flaw.

When it received mail, it renamed the file and changed the ownership label. That meant you could move a file into any part of the setup, even protected parts. A hacker could mail a program into the protected part of the system, where it could later start running. When it ran in that protected space, there were no limits to what the program could execute. It could even make a hacker into a super user.

When this hacker discovered that he could become a super user on the LBL computers, he must have just felt like he'd fallen into a tub of butter. A super user, again, could steal files, could cover up tracks. It was like becoming invisible, capable of getting away with any crime. It would have been easy enough for Stoll to patch the hole and lock this hacker out. That would have been the easy road. But Cliff Stoll had noticed something else. This hacker wasn't just poking around the Berkeley lab. They were leapfrogging from their network into the mill net,

the network used by the U.S. Department of Defense. You could do anything you wanted to any file on the computer, including download something. Really, nothing could be protected from a super user. So what was this intruder doing with all those military files? Stoll decided it was his job to find out. What's more, he decided it was his This was a new kind of crime, and Cliff Stoll had a hard time convincing anybody they should be worried. If you go read his memoir, The Cuckoo's Egg, you'll be amazed at all the back-and-forthing he did with the FBI and CIA,

just trying to get their attention. At one point, he asked his boss, what cops are in charge of the ARPANET? His boss said, damned if I know. Stoll would have to do it himself. He'd have to lie in wait, learn to identify the intruder, and trace them back to wherever they were based. He'd have to catch the invisible intruder himself. The intrusion took over his life for several weeks. Richard Baitlick is a strategist and author-in-residence at the security company Corelight. I don't think Cliff Stoll knew it at the time, but he was basically inventing the way people investigate intrusions. Over those first weeks, Stoll set up a system that alerted him via his pager

whenever Sventec logged into the lab's computers. Then Stoll would race to turn on his printers and watch the hacker's activity. He was doing a painstaking kind of intrusion detection that today would simply be automated. Each time Sventec showed up, Stoll tried to trace it too. Eventually, he pieced together that the hacker was coming in from overseas. The calls were coming via a satellite called Wester 3. And finally, after many weeks of wrangling with authorities, he had narrowed things down to West Germany. That's where he hit a major roadblock.

The German phone exchange, which the hacker was being routed through, was built in the 1950s. That meant, in order to trace the call, technicians had to test every switch by hand. It would take over an hour. Making things worse, the hacker was still using the Berkeley lab as their main launch point into U.S. military systems. But they just never stayed on the lab's computers for that long anymore. Stoll had to find some way to keep the hacker busy while the Germans did their work.

And so his girlfriend at the time came up with the idea of providing fake documents that these intruders might be interested in. That girlfriend was Martha Matthews. She told Stoll, if the hacker is looking for interesting secrets and seems to be interested in the military, why not just give them a treasure trove? Every time that they were logging into the network, they were looking for defense-related information. And so Cliff and his girlfriend created all of these fake documents involving something called SDINet, SDI referring to Strategic Defense Initiative, which was the official name for what a lot of people remember as Star Wars under President Reagan. And they just invented all of these fake documents. Stoll was creating what we'd call a honeypot, a lure for criminals.

He uploaded these reams of fake secrets to the lab system. And then he waited. The hope was that if the intruders found this information, not only would they be interested in it, they would spend hours downloading it. And that's what happened. At last, the German technicians, scrambling away in their 1950s phone exchange, were able to trace the hacker to the city of Hanover. And then to a single individual,

a man called Marcus Hess. He was in the early generation of hackers who just considered it a challenge to break into computers. Katie Hafner says that Marcus Hess probably began his attack on the Berkeley lab without much malicious intent. Don't forget that programming is really all about solving puzzles. I think that he thought it was pretty special to see that he could dial into a computer in California. I mean, there you are, you're basically someone in your 20s, you're working for this startup in Hanover. The world feels very small.

And suddenly the world opens up to you. It may have begun as a puzzle. But Hess was not alone over in Hanover. He was actually part of a club of hackers. They are the ones who brought him into their little hacking ring. They had met up with some Soviet... You know, this was back in the 1980s when the Cold War was really in full swing.

And there was a lot of worry about what could happen with classified information. Don't forget that this was in the era of Star Wars. And so the Soviets would have wanted any information they could't forget that this was in the era of Star Wars, and so the Soviets would have wanted any information they could get about that. So Hess found himself hacking for info that could be sold to the Soviets. They pulled him into it. Hess was a Unix guy. The other hackers didn't know Unix, and this mattered because the Lawrence Berkeley lab was running on it. So when that hacking ring brought on Marcus Hess, one of the unintended consequences was that somebody was now in a position to break into the very lab where Cliff Stoll had come to work.

Six months after Stoll was told about that 75-cent discrepancy in his lab's accounting, the German police arrested Hess, along with others from that hacking ring. Stoll flew to Germany to testify against him. Hess was found guilty of espionage and given a 20-month suspended sentence. Something vital had been proven. Cliff Stoll showed the world that sensitive government secrets were vulnerable to computer hacks. A whole landscape of international digital espionage snapped into focus. The FBI, the CIA, every agency was finally paying attention.

And that's when people started to get serious about building better defenses. Today, intrusion detection systems and honeypots like the one Cliff Stoll made have become part of everyday security. But we have to remember what a breakthrough these concepts really were. A friend that was working in cyber recommended it to me. Read this book, and I was immediately hooked. Don Cavender is a retired FBI cyber agent, and he's talking about the book Stoll ended up writing. Stoll's story has helped shape the entire field of security. It's amazing that his book is still one of the top recommended reads for somebody new coming into cybersecurity. And he had no cybersecurity background whatsoever. The firewall hadn't been invented yet.

So there was no perimeter security at all. There was no type of network security at the time. Stoll had to figure it all out on the fly. Cliff Stoll not knowing any better, what he did was he let the guy in. He kept him online. He kept the hall open. He monitored everything,

worked on tracing things back, and would reach outside of the lab, basically make noise, trying to tell the right people what's going on. He went totally against what everybody else was doing at the time in network security. But after Stoll's work, the field began to evolve. For one thing, we realized that in a networked world, everybody has to take part in making things secure. Just because you don't think anybody wants what's on your computer

doesn't mean you can't be a gateway to something more sensitive. This is something that can actually happen. If you're vulnerable, you could be used as this jump-off point on the way to somewhere else. Authorities eventually learned that Marcus Hess had broken into about 400 U.S. military computers. Beyond that awareness of our shared danger, though, Stoll also showed that cybercriminals are always going to make use of brand-new possibilities, brand-new vectors of attack. That means security teams have to be constantly reinventing their jobs. They have to stay creative.

Now we use more sophisticated means to kind of trace back those types of communications, but he used what he had at his disposal at that time, which was a lot of creative thought on his side. Cavender said he was inspired by Stoll's story, and he wasn't alone. In researching this episode, we heard from many people in security who also read The Cuckoo's Egg and felt called to the world of digital security

that people like Stoll were just beginning to uncover. That may be his greatest legacy, not just the technical tricks he thought up, but the philosophy that a single person can use the tools at their disposal to make the world a little more safe. I'd say there's a whole industry full of unsung heroes out there that will never get recognized that are day-to-day saving civilization through their efforts. At the trial in Germany, one of the arrested hackers told the news cameras, At the beginning, I wasn't thinking of anything. I was just sitting there, hacking. At another point, Cliff Stoll says, when he ran into one of the hackers in the courthouse washroom, the hacker complained to him that his life was being ruined. Stoll could only shake his head.

It seemed like none of these guys understood the severity of their actions. Like the horror film character, The Invisible Man, those hackers thought they could snoop and steal without any repercussions, that they would never get caught. But some of the foundations of our intrusion detection systems were invented by Stoll

right as their hacks began. I'm Saranya Tbarik, and this is Command Line Heroes, an original podcast from Red Hat. Next episode, it's our season finale. We've got the tale of a little program called Satan. That's honestly its name. And like the biblical Satan, this particular program managed to create a lot of chaos once it was unleashed.

You'll want to hear how that drama plays out. Subscribe, follow, wherever you get your podcasts. And until then, keep on coding. Hi, I'm Jeff Ligon. I'm the Director of Engineering for Edge and Automotive at Red Hat. The number one hesitation that I think folks have about Edge is that they assume they're not ready for it. They think that it'll mean starting over from scratch, or that every time their business needs to change, they'll be re-engineering solutions and re-evaluating vendors.

And look, Edge is complex, and it does mean a different approach, particularly for devices at the far edge. But with Red Hat, it doesn't mean starting over with new platforms. We've taken the same enterprise solutions that simplify and accelerate work across the hybrid cloud and optimized them for use at the edge. So you can manage your edge environments just like you do the others. Come find out how at redhat.com slash edge.