CyberWire Daily - That first CVE was a fun find, for sure. [Research Saturday]

Episode Date: November 14, 2020

In the late 90s, hackers who discovered vulnerabilities would sometimes send an email to Bugtraq with details. Bugtraq was a notification system used by people with an interest in network security. It was also a place that might have been monitored by employees of software companies looking for reports of vulnerabilities pertaining to their software. The problem was - there wasn't an easy way to track specific vulnerabilities in specific products.  It was May 1999. Larry Cashdollar was working as a system administrator for Bath Iron Works under contract by Computer Sciences Corporation. Specifically, he was a UNIX Systems Administrator, level one. His team managed over 3,000 UNIX systems across BIW's campuses. Most of these were CAD systems used for designing AEGIS class destroyers. This position gave me access to over 3,000 various flavors of UNIX ranging from Sun Solaris to IBM AIX. Joining us in this week's Research Saturday to discuss his journey from finding that first CVE through the next 20 years and hundreds of CVEs is Akamai Senior Response Engineer Larry Cashdollar. The research can be found here:  MUSIC TO HACK TO: MY FIRST CVE AND 20 YEARS OF VULNERABILITY RESEARCH Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. of you, I was concerned about my data being sold by data brokers. So I decided to try Delete.me. I have to say, Delete.me is a game changer. Within days of signing up, they started removing my personal information from hundreds of data brokers. I finally have peace of mind knowing my data privacy is protected. Delete.me's team does all the work for you with detailed reports so you know exactly what's been done. Take control of your data and keep your private life Thank you. JoinDeleteMe.com slash N2K and use promo code N2K at checkout. The only way to get 20% off is to go to JoinDeleteMe.com slash N2K and enter code N2K at checkout. That's JoinDeleteMe.com slash N2K, code N2K. Hello, everyone, and welcome to the CyberWires Research Saturday.

I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. One of the PR guys, Tim, he thought it would be interesting if I spoke about my first CVE since I had been at Akamai for 20 years. It had more of a story behind it than I think Tim was expecting, so it was actually caused some mayhem. So I wrote it up and here I am. That's Larry Kashtaller. He's a senior security response engineer at Akamai Technologies. Today, we're discussing his recent blog post, Music to Hack To,

my first CVE and 20 years of vulnerability research. And now a message from our sponsor Zscaler, the leader in cloud security. Enterprises have spent billions of dollars on firewalls and VPNs, yet breaches continue to rise by an 18% year-over-year increase in ransomware attacks and a $75 million record payout in 2024. These traditional security tools expand your attack surface with public-facing IPs that are exploited by bad actors more easily than ever with AI tools. It's time to rethink your security. Zscaler Zero Trust Plus AI stops attackers by hiding your attack surface, making apps and IPs invisible, eliminating lateral movement, connecting users only to specific apps, not the entire network, continuously verifying every request based on identity and context,

simplifying security management with AI-powered automation, and detecting threats using AI to and AI. Learn more at zscaler.com security. All right. Well, I mean, let's dig into it. Can you sort of set the scene for us? So what era are we talking about and where were you in your career at the time? So I was studying computer science in the U.S. of Southern Maine back in 1994, 95 era, and I was working for this small company in Southern Maine, in Portland, Maine, actually. And I was there as an internet analyst, and I ended up getting a position at Computer Sciences Corporation. don't know, Computer Sciences Corporation is a large consulting company where they hire folks to take care of their IT infrastructure. So at the time, Bath Ironworks was contracted or had contracted CSC to handle their IT infrastructure. And I was looking to get more of a broad view of the internet world and the computing world. And CSC also paid more. So I got hired at CSC.

And the playground I had was what I called it was they had my team manage over 3,000 Unix systems. There was anything from SGI machines to IBM AIX machines to HPUX machines. So I had this enormous pool of Unix systems to play with. And the first day on the job, my manager was giving me a tour of various buildings around Bath Ironworks because Bath Ironworks has a large campus. Bath Ironworks, you know, because Bath Ironworks has a large campus. And he took me into one of these rooms where they had, I can't remember if it was a dozen SGI indie boxes. And they were all sitting there humming along and they were working on a new submarine program back then. And this was a room that they were going to start, you know, doing this development work and with 3D images. My manager sat down at one of the consoles

and he says, you know, someday when you prove your worth, I'll give you a login on this. And me coming from a security background, I knew that IREX systems had LP account and that LP account had no password. It had a valid shell, but you didn't need a password to log in.

So you could simply walk up to one of these machines type lp and hit enter so what i did was i i strolled up to one of the machines and i typed lp and i hit enter and logged in and i looked at him and i said thanks i don't need one and he looked at me and his mouth dropped and he's like how did you do that and i said by default irx6x machines don't have a password set for LP. You know, none of these machines have passwords set on that account, so you can literally log in to anyone who's an LP user and get a full desktop. So he looked at me and he said, would you do security for us? And I said, I was hoping you'd say that.

So from then on, I was sort of like the penetration tester slash hacker kid that was doing security testing. And one of the story goes is that we had this SGI Onyx 2. For folks who don't know, this thing's about the size of a refrigerator. Back then, it would cost between anywhere from $250,000 to half a million dollars, depending on how it was configured. This one that we had had its own private room that it sat in with the raised floors and the air conditioning. It had a punch key code on the wall. You had to punch in a number that only a handful of sysadmins knew. None of my team knew because we were only sysadmin level one.

Only the sysadmin level threes had access to it so if you needed something from that room you had to ask them to go get it for you they wouldn't give you the code uh you know you had to ask someone hey can you go in this room and get me a new hard disk or whatever you needed and they'd go in and get it for you and uh the sysadmin who dave i think his name was who had access this room, would taunt the rest of us, you know, the guys in my group, like, hey, you know, someday maybe you guys will get root on the Onyx machine or you'll get an account on the Onyx machine. And I'll give it to you, you know, once you're maybe at my level of sysadmin.

So, you know, this sort of bothered me. Let me interject here, Larry, because for folks who aren't from this era, who may be unfamiliar with these Silicon Graphics Onyx machines and the types of machines that SGI was putting out at the time, if you were working in graphics or 3D animation or really any of these things that you could apply one of these machines to, the type of processing power that you would need for one of these things. These machines were objects of desire. Not only were they extremely powerful, but they were beautiful machines as well. If you go look up a picture of one today, you'll probably say, oh, it looks like a computer from the late 90s or the early 2000s.

So they look a little dated by today's standards, but take my word for it. At the time, these were extraordinarily sexy devices. Yeah, they were, they were the creme de la creme back then. Yeah. So, so to have access to the Onyx 2 was like the Holy grail for my team and mostly me. I don't think the other guys really cared but i i just didn't like being taunted so i thought to myself you know i already know that i can log into that system with the lp account and get a shell on it because i know they didn't secure it but i want to get root on this thing so how can i how can i get root on the onyx so i'm like well i can i have another sgi system that i have access to that's near me in my lab, in my office that I had, I can log into that system and just look around the operating system.

And back in the 90s, when you attack the system, you were trying to get hacked into the server and get root. You were trying to get root so you can wipe your logs and your access showing that you had logged in and hide your tracks. And that was the big thing to get root on a system was for black hats. And so I knew, you know, what I should do is look for set UID root binaries, which is binaries on systems that when you execute them as a normal user, they execute with root privileges.

And I figured if I could find a binary with set UID root permissions that I could somehow abuse to get it to do something that it wasn't meant to do, like write to a system file or execute shell as root, I can get a root prompt on the machine and I would be all happy and have hacked into the system. So I saw this looking around user S bin where most of the set UID root binaries lived back then. I saw this file called MIDI keys. Now, MIDI keys had the Setuid bit was set on it, and I had never really heard of it before and was curious why, what this binary was that needed a root access. So I ran it, and on my screen pops up this little keyboard,

looks like a piano, and when you click the keys, it plays these little tunes, these little midi tunes and you can save them you could you could compose a little midi um song and save it to disk and i thought to myself well can i edit files with this can i can i save to a file as root and possibly edit the password filed. So I opened up, open up the password file, added, I put a zero in for my user ID for Larry and then saved it and then logged in again. And I was rude.

I'm like, I found a, a way to get root on the Onyx. So I quickly log in the Onyx as LP forward and X window back to my machine and forward back a execute the MIDI keys, bring it up, open up Etsy password, create a Larry account, make my user ID zero, and then save it, log back into

the Onyx and I have root. And I'm like, okay, I've hacked into the Onyx. I have a root prompt now. I should set the password on the Onyx so it's not sitting there with a Larry login with no password. It's super insecure. So I type password to change the password. I'm like, oh, no, my user ID is zero. I'm changing the password for root, not for Larry. So I go to back out and I hit control D, control D, thinking that I would back out of the password program. And instead, the IREX system saved my password as control D. So I changed the root password on the SGI Onyx 2 to control D. At this point, I'm like dreading it.

I'm like, I turned white and I'm starting to sweat. And I'm like, shoot. I'm like, I got to go tell Dave, the system man, who doesn't like me very much, that I just changed the password on the Onyx to control D and he's not going to be able to log into it. So I asked my friend Donovan, who is much bigger than I am. I'm small and five foot six. He's six foot three. And I'm like, Donovan, I'm like, can you go tell Dave that I, you know, change the root password on the Onyx system to control D and if he could, you know, fix, fix it. And, um, so Donovan was like, okay,

sure. And Donovan came back like 15 minutes later and he's like, dude, he's like, they are so mad at you. And I'm like, wow, what happened? He's like, when you change the password, they were in the middle of giving the Navy a demo of the 3d modeling on the Onyx two. And he said, there was an Admiral standingiral standing there there were there was upper management sitting there and they couldn't log into the Onyx and I'm like oh my god I'm like I'm gonna get fired I'm like son of a I'm like that's it I'm like I'm gonna lose my job so I you know Donovan came back a little while later he's like he's like you know Dave managed to change the password he had a hit a prompt or a login open on his desktop to the onyx so he changed the password to back to the original and i'm like

okay i'm like my my manager's gonna be here you know in a half an hour and tell me it's i'm gonna get fired and lo and behold half an hour passes and my manager shows up he's like i need to meet with you with my manager in his office and i'm like cr, ah, crud. I'm like, here we go. So I go to his manager's office and I notice on the wall, it has a San security poster. And I'm like, this guy knows about security. He's aware of the San security conference. And it has all of the posters all decked out with all of these security procedures and things that do hard in your system and stuff like that and the newsletters that you can get from SANS. And I'm like, cool. I'm like, okay, well, to tell me your manager and the person,

the sysadmin who your machine you're testing, that you're going to be doing it and when. He's like, we can't have an incident like this happen again. Example, the Onyx 2. And he says, you disrupted an important demo. He's like, but I understand security is important. He's like, I've been talking to these guys about security. And he's like, I'm glad that you're actually out championing it, but I don't want you to actually break systems and bring systems down. He's like, we can't have systems break because you're testing security. He's like, you have to do it in a safe manner. So he's like, you need to let me know. And at this point, are you thinking he said again? He said again. He said he's referring to the future. Yeah, my brain is going, you still have a job. You still have a job. He said again, that means you're here.

And then, you know, he's like, so from now on, he's like, I want you to email me with a report of what you're testing, when you're planning to test it, CC your boss. And he's like, CC the people that you're going to be asking to test. He's like, if they give you permission to test, he's like, CC, the people that you're going to be asking to test is like, if they give you permission to test, he's like, you know, you can go ahead and test, but he's like, you have to let them know. And they have to say it's okay, because you can't test when they're trying to do something and you mess it up. And I'm like, okay. And he's like, you know, have a nice day. And I'm like, okay. So I walked out of there and I'm like, great. I'm like, I have legitimacy now. You know, he gave me a framework that I could work in to tell people, hey, you know, your systems are getting a penetration test this week and you need to pick a day and a, but it did get them to secure the system before I tested it.

So it was a win-win. They actually took security more serious, and then I still got to test the system. Now, it may not have been easy to get into, but at least it was fixed. So I lucked out. It was a good opportunity and thing that had happened after I'd gotten into that trouble. Wow. So it was fun. So where did that ultimately lead?

I mean, what's the sort of the connecting dots between that incident and where you are today and this bigger conversation about CVEs in general? That sort of was like my first taste of blood. You know, it was like, I really enjoyed finding a vulnerability and finding a way to exploit it. And then, you know, getting access that I wasn't supposed to have. And it made me like, I like to solve puzzles like that. So I started doing more, like before I had gotten that or found that vulnerability, I didn't know how folks found vulnerabilities. It was, you know, I was studying them. You know, the folks that I worked at with a company would always study these new vulnerabilities that came

out and wonder, how did this person know where to look? How do they find this? And, you know, and then I realized you just have to look, you know, if you're not looking, you're not going to find anything. If you're looking, eventually you're going to find something and you know and then i realized you just have to look you know if you're not looking you're not going to find anything if you're looking eventually you're going to find something you just have to know what to look for you know i knew what to attack i knew to look at security root binaries so i managed to find something that somebody hadn't looked at or found before so that sort of gave me the the the mindset and the idea that if i look at other software and look for common programming problems or mistakes, I might find more vulnerabilities. And then that's what I've been doing for the last 20 years when I can. Did you ever have any more insights into that little MIDI keys program

itself? I mean, just for our listeners, MIDI is a protocol that musical instruments use to communicate with each other. And back in the late 90s, it was not as ubiquitous on computers as it is today. I'd guess that most systems today come with some sort of MIDI implementation just as a matter of course, because it's all over the place. implementation just as a matter of course, because it's all over the place. But I'm just trying to imagine, you know, the folks who made this SGI the size of a refrigerator, you know, did there just happen to be a developer on that team who had a side hobby of hooking up, you know, playing with electronic keyboards at the time, you know, a Yamaha DX7 or something. And so that's why this was on this machine.

Like, what was it doing there? That I don't know. I assume it was just, you know, another little bell and whistle to add to the Irix portfolio of neat things you can do with the system. And, you know, it just, it never occurred to me that there would be this music program on this operating system that was pretty much meant for 3D modeling. I didn't expect it, which is why I had to see what it was, because it just seems so out of place. Which is an interesting lesson in itself, right? Right.

And then I later found out on BugTrack, after I had published my findings to BugTrack, that some of the folks on there said if you change the editor in your environment, your environment variable editor from Vi to bin shell, and you launch the editor from MIDI keys, it just spits out a root prompt. So you didn't even need to edit the password file. You could just pop out a root shell because it executes the Viya editor as root. So it was even a quicker way to get root. So it was, yeah, it was, you know, and I'm not sure why it was there

and then why it was set UID root, but it was a fun find for sure. Yeah. Well, let's fast forward to today. I mean, you and I have spoken on this program and other places many, many times. You have a number of CVEs under your belt. What's the latest from you? have, I think I have only about 200 and something documented, but there are folks like I have a friend who had started IBM X-Force and he had been tracking my vulnerabilities from day one. He actually mentioned to me over Twitter that he's like, I remember when you found that vulnerability and it's like, oh yeah, you were, you were running the database over at IBM tracking all these

vulnerabilities. So of course you remember. So, uh, so uh so it was um it was just something that i i had done and then now it's uh well i i guess i got a bunch from wordpress plugins because i had done a little experiment there but you know and then now it's more web application stuff is what i've been attacking more web application and then uh i still have a soft spot in my heart for temp vulnerabilities. I recently found a slash temp race condition vulnerability, or it was for Solaris 11 x86, where one of their utilities would create a file in slash temp. And then it would chamod that file to be world writable. So you could use that to chamod etsy shadow and change the password and get yourself to root. So I got a CV for that like two months ago.

And then I have two more, I think temp vulnerabilities coming out next month for Oracle Solaris 11. So yeah, I don't know. It's, it's, it was just fun. Yeah. Well, you know, when you look back on, on this incident in incident in particular, you know, as a way, the sort of, you know, you got your start. It kind of set the hook for you. It set you on this path, right? How has that informed the way that this played out and how it informed the way that you look for these sorts of things? informed the way that you look for these sorts of things?

I mean, do you think there are dots to connect all the way back to that, that that really sets you on this path and you still approach things in a similar way? Yeah, I think that, I don't think if I had ever found that vulnerability, I don't believe I would be where I am today because that was a spark for me to say, okay, this is, you've done this once. You now know where to look and what to look for. And you were able to find something just from looking. You can keep looking and find more things to find security vulnerabilities in. And from then on, I was always interested in finding security holes. And I remember in early 2000s, I would open up trade magazines and look through them for software that ran on Linux that you could get a free demo.

And I would download the demo and I would look for vulnerabilities. And I would find, you know, either temp vulnerability where there's race condition and you could elevate privileges to root. Or, you know, there were some where they had there was code execution errors and things like that. So so that was how I used to find vulnerabilities then. And then working in the Akamai CERT in the last couple of years when I had joined there, my then boss was like, you know, he's like, do you like breaking stuff? He's like, you should really look at WordPress plugins. Those things are full of holes. And I'm like, OK.

He's like, you should really look at WordPress plugins. Those things are full of holes. And I'm like, okay. So I started looking at those and found lots of vulnerabilities in those because there were just so many. It didn't really have a process for checking the security or the coding requirements for those plugins. So I ended up finding a lot of stuff with that. And I did a bunch with RubyGem. So it was just sort of like a, I don't know, like a prairie fire for me. That was the initial match.

You know, I think something you and I have in common is, you know, those of us who remember this era of computing, I think most of us have a certain amount of affection for it. You know, there were things that were harder than they are today. But I think it's easy to look back on those days fondly. And I'm wondering, you know, for folks who are getting started today, who the folks who are, you know, perhaps we have students listening to us or folks just starting out in their career, who knows what the next 20 years is going to hold for them

in terms of the changes that they're going to see, you know, similar to the changes from 20 years ago. And I'm wondering, do you have any insights there on the, you know, how those things from long ago are still informing the work you do today? Any tips or words of wisdom for the folks who are just starting out? Just be, you know, don't be afraid to fail and, and really just realize that you're, you're going to learn from your failures and you really want to just learn how to just Google stuff that you don't know, search for things. And just, you know, if you find something that you're interested in, try and find out as much as you can about it and try and look for things that could benefit your if you're looking to find a vulnerability.

And I guess what I'm trying to say is, you know, learn as much as you can about that thing and then don't lose your your your traction for for learning new stuff. I'm always learning new stuff and it gets more tiring as you get older. But in your 20s and 30s, you know, you should just try and absorb as much as you can. And always check with the boss to make sure there isn't an admiral on site getting a demo before you do your stuff, right? Yeah, don't do anything illegal. So make sure you've got permission to do stuff. Our thanks to Larry Cashdaller from Akamai for joining us. His blog post is titled Music to Hack To,

My First CVE in 20 Years of Vulnerability Research. We'll have a link in the show notes. Cyber threats are evolving every second, and staying ahead is more than just a challenge. It's a necessity. That's why we're thrilled to partner with ThreatLocker, a cybersecurity solution trusted by businesses worldwide. ThreatLocker is a full suite of solutions designed to give you total control, stopping unauthorized

applications, securing sensitive data, and ensuring your organization runs smoothly and securely. and ensuring your organization runs smoothly and securely. Visit ThreatLocker.com today to see how a default-deny approach can keep your company safe and compliant. The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliot Peltzman, Puru Prakash,

Stefan Vaziri, Kelsey Bond, Tim Nodar, Joe Kerrigan, Carol Terrio, Ben Yellen, Nick Vilecki, Gina Johnson,

Bennett Moe, Chris Russell, John Petrick, Jennifer Iben, Rick Howard, Peter Kilpie, and I'm Dave Bittner. Thanks for listening.