CoRecursive: Coding Stories - Story: Apple 2001

Episode Date: April 3, 2021

David Shayer worked at Apple for 14 years, and he has a wild experience to share. Apple has a unique culture, and David will give us an insider view of what it was like for him at Apple during the 2000s, roughly between 2001 to 2015 when Apple transformed into the powerhouse that it is today. David worked as a Software Engineer but for the hardware organization with Apple. He worked on a few special projects at Apple: at least one of them was top secret. But he is also going to share the struggles of building file systems and working on really short timelines and having development plans upended by Steve jobs. Episode Page

today on Code Recursive. So I was in my office one day and my boss's boss came in and he said, have a special project and I want you to work on it. There's some guys that are going to come over from the US government and we're going to help them build some kind of device. And I said, what do you mean? And he said, oh, I don't know anything more than that. And he said, your boss doesn't know anything about this. He just knows you're working on some special project. Don't talk to him. Only talk to me. But didn't that make you suspicious? Apple had a lot of special projects that would go on at various times.

You know, there were people who would have their office doors closed. And, you know, whenever you would go and see if they wanted to go to lunch, they would actually have black cloth. They would put over whatever they were working on. This was not that uncommon. Hello, and welcome to Co-Recursive. I'm Adam Gordon-Bell. Each episode of Co-Recursive, someone shares the fascinating story behind some piece of software being built. Today on the show, we have David Shearer. He worked at Apple for 14 years, and he really has a wild experience to share. He's going to give us the insider view of what it was like for him at Apple during the 2000s, roughly between 2001 to 2015, when Apple transformed

into the powerhouse that it is today. There will be special projects, some of them top secret, there'll be government agents, but also just building file systems and working on really short timelines and having development plans upended by Steve Jobs, and also just generally working hard under high pressure. The story starts in the summer of 2001. I was an independent developer writing Mac software and selling it to companies that wanted things published and didn't have enough engineers. And there was an Apple poker game that Apple engineers ran on the side. The poker game originally took place actually at Apple in a conference room. There was one

conference room that had a nice round table that was just the perfect size. So everyone would show up like at 7 p.m. And most of the people were Apple employees, but there was a Google employee and a Microsoft employee and, you know, a couple independent people like me. So you would just like buzz in or something? Somebody would come? Yeah, you'd call, you'd call someone and say, Hey, I'm downstairs. And someone would come and get you and take you up. I got myself invited to that poker game and would go every week and lose my 20 bucks. So they'd keep inviting me back and would keep my ears open because, you know, you could find out all kinds of interesting stuff about what was happening at the company and you could get tech support

occasionally. You know, you'd say, hey, you know, I'm having trouble. You know, this call doesn't work the way the documentation says it's supposed to work. And, you know, someone at the poker game would usually say, oh, yeah, I wrote that. The documentation is wrong. You need to make this other call first and then it'll work. And, you know, this would save me days of messing around. It was like the cheapest, most reliable form of tech support available. And it was fun. You know, I like these guys.

They were friendly. And one of the things I was doing at that time was working for Symantec, working on Norton Utilities, which was a disk repair program. And so I was deep in the Mac file system, repairing broken catalogs and such. And I got a call from one of the guys at the poker game that said, Apple wants to hire someone who knows how to write a Mac file system. It's kind of a specialty, right? There aren't a lot of people who know file systems well. And I've been doing it for years. And so he said, call this guy, Tony Fidel, who I'd never heard of, and tell him you want an interview for this open position. So Tony knew I was going to call. I called Tony and said, hey, I hear you need a guy who can write file systems. So I came in to interview. What did you think going in? Were you

like nervous? Were you hopeful or? This was in 2001 and the dot-com bubble had just collapsed in the stock market, which meant most of the companies in Silicon Valley that had been giving me work, you know, had either gone bankrupt or even if they weren't bankrupt, they had canceled everything that wasn't actually earning them money that day. Right. And so all my contracts disappeared like within a couple of months. I had no work. Right. It was the first time since getting out of college that I didn't have any work. Nobody was answering my phone calls at any of these companies. In fact, most of the people I know that have been laid off. So it was pretty bleak. So when I heard there was a job, I'm like, hell yeah, I'm taking that.

And then when I heard it was to write a file system, that was perfect. The Mac file system was called HFS Plus, which was proprietary to Apple. And there weren't a lot of people who knew a lot about it. Probably half of the people who knew how it worked, worked at Apple already. So obviously I knew that those guys were all busy or else they would have just assigned them to work on this project. And the other people who knew about it, there were probably a couple dozen out in the world. Not all of them lived in the US. Most of them were doing other stuff. So I knew that there wasn't a lot of competition for this rather particular skill they needed. So I came in to interview and Tony said, can you write a file system? I said,

yes. And he said, you know everything there is to know about Mac file systems? I said, yes, pretty much. I've been working on them for years. I know an awful lot about it. And then there was kind of a pause and I realized he didn't know anything about file systems. He didn't know what else to ask. He was an electrical engineer, a hardware guy, and he was really smart and really good at what he did, but he wasn't a software expert. So basically just said, okay, can you start on Monday? And I said, sure. What are we building? And he said, you don't need to know. So David heads into his job, not knowing what he's going to be working on, but he does know that the guy that hired him was an electrical engineer and that he's going to be working for

the hardware division of Apple. So Monday I show up at Apple and they show me to a cube and there's a development board. And the development board looks like, you know, the motherboard from a desktop PC that had been taken out and put, you know, on my desk because it has all the components out on the board and it's got a test point. So you can hook up oscilloscopes and logic analyzers and stuff if it's not working right and diagnose the problem. It's got a big power supply off on the side to power.

It's got a big full-size desktop hard disk hooked up on the side. This is pretty much the way all new computers are developed. They start out like this so that in the early days, hardware engineers can hook test equipment up and see what's not working. And then as you get into production, you get the tiny test equipment up and see what's not working. And then as you get into production, you get the tiny little board, which ends up in your laptop. So it's like a foot by a foot, and it's got a bunch of discrete components on it. And the SoC with the dual ARM cores in the middle and hardware test points all around the outside of the board. Apple's computers at this time were based on the PowerPC CPU,

but this machine sprodoed all over David's desk has an ARM CPU, and he's given a Windows computer. This is really strange, but he knows what he was brought in to do, to get a file system up and running on it. So I used my Windows machine with ARM development tools, because someone at the beginning decided we were going to use the Windows version and not the Linux version of the ARM development tools. And you would compile the operating system in C++.

You would load it on across a little serial interface. And then you would run it and you would debug it with this little debugger that went across a serial debugger connection called a JTAG, which stands for something. I don't know what it stands for. That's crazy. Like I would have been scared if I came into a new job and they had like what appeared to be like a motherboard hooked up to a serial cable, which you don't even see those anymore. And then like an oscilloscope or whatever, like I would have been worried that

I got the wrong job or something. Well, I didn't have an oscilloscope in my office because I'm not a hardware engineer. I wouldn't know what to do with it. But we had the same boards that the hardware engineers had because that was the only board that was in production. At some point, you know, the production line gets up and they start doing test runs. So like every week they do a test run and they produce a couple dozen boards and they get shipped from China back here for us to play with them and see if they're working. Right. And they don't want this stuff just sitting out, you know, unprotected where people can't see it. So they make what Apple calls a stealth case, which is just a big plastic box. So our stealth case was so ugly, it looked like they went into an old Russian medical equipment leftover warehouse and just took some plastic boxes and stuck it in. It was horrible. They said, basically,

you're writing the file system for this thing. You don't need to know what's on the disk. Just make it work. So David dives into making it work and they start to build up a team. I was the second software engineer hired. They were hiring quickly, but still, this was not a big team. There were probably a dozen people total when I started, you know, a couple of hardware engineers, you know, a mechanical engineer, a product manager. It wasn't a big team. By the time we shipped at the end of the year, there were probably two dozen people. So it was a small team. And the first meeting that I went to, they said, we're going to ship

by the end of the year. And I thought it was a joke because I thought nobody can ship brand new product with new hardware and new software in, you know, like six months. And I was about to laugh because I literally thought it was a joke. And I noticed no one else is laughing. So I'm like, oh, this is what I signed up for. So we were working, you know, like literally 80 and 90 hours a week, seven days a week till midnight or later to get this thing out. I remember I was out on a Sunday night in September with my wife at a concert. And like at 10 PM, my boss calls me at the concert on my cell phone and says, there's a bug and we think it's in your software and you need to come in here because, you know, when the guy who is going to call this code, you know, comes in Monday morning, it needs to work. So, you know, I told my wife, you know, when the concert's over,

I'll drive you home and I'm driving into work. And I went in and spent Sunday night in work fixing it so that, you know, Monday morning it was ready for the guy who was going to be the customer of this API. The operating system for this mysterious project wasn't being made from scratch. The lower level parts of the OS had been licensed from a third party. Apple's file system was called HFS Plus. And the people that we bought the lower level of the operating system from obviously didn't know what that was, hadn't supported it because Apple was not taking over the world back in 2001. So they didn't see any reason to support Apple's file system. They supported FAT32 because that was what most people used back then.

I took the HFS Plus code from the Macintosh operating system. The first problem I had is that the Mac at that time was a big Indian, and the ARM chip was a little Endian processor. And the file system has all these data structures that are written out to disk, and they're written out big Endian. So we had to add code to reverse all those as they come in and as they come out. Big Endian and little Endian is like the order that numbers are written in binary? Right. When you take a number, like an integer or a long, it's the order that the bytes are actually in memory, right? In Big Endian, the most significant bit

is at the lowest byte number, and in Little Endian is the reverse. This was the first time that I was aware of Apple had shipped a little Indian processor. And so I had to put in code to handle that, which is not difficult. It's mostly just a very detail-oriented matter of finding like every single data structure in the file system. And of course, the file system has variable size data structures, right?

It's not like just simple data structures. A lot of them are variable size. The development loop was terrible, though, because the ARM development tools were slow and loading your code across the serial cable was slow and the debugger was terrible. So, you know, how big this field is depends on these other fields that have come before it. So there's a lot of calculation and there were various places where, you know, I'd miss something and then it would crash and then you have to go back and figure out, okay, what did I miss?

Like we already have rules for how we write down numbers. Why not follow the lead of like humans? Follow how decimal works. I think the hardware engineers told me once that you use one less gate if you do it little Endian rather than big endian. And so Intel, and I guess ARM chose little endian because it used slightly less hardware. Fortunately, when the TCP IP was developed for the internet, they picked the standards. And my memory is that all stuff sent across TCP IP is big endian.

Yeah, that's the one that makes sense. Yes, it's the one that makes sense. If you just take a hex editor and you look at the memory just in straight hex, Big Endian is the one that looks like you expect. And Little Endian is the one you were like, okay, this byte actually goes on the left of that byte. And then you're doing all these corrections in your head. I have to be honest, I've never had to use a hex editor in any sort of software development task, but David has. And so as he works through this kind of monotonous task of mapping the byte orders back and forth from one style to another, the secrecy on the project just continues. We weren't on the main campus. We were a quarter mile away in an accessory building,

which mostly had people who did back-end work at Apple, the people who ran the computers that did accounting and logistics and shipping and stuff like that. So they just took part of a floor in that building. They put up security doors so those guys couldn't come into our area, and they put us in there. They were so paranoid. They told us, when you go to lunch, because we would go to the main building

with the main cafeteria at lunch, they said, you can't all walk together, right? You can go in groups of two and three, but like all of you can't go to lunch together because we don't want the other Apple employees to look and say like, oh, who are these people? We've never seen them before. What are they working on?

That's crazy. Yes, they were serious about it. It seems like a vague vibe of like paranoia that I'm getting across from your stories. I wouldn't call it paranoia, but Apple definitely has a lot of secrecy. I mean, people have always been interested in what Apple is doing, right?

There's all these websites which are printing, you know, rumors about what's going to be in the next iPhone, you know, or what's been added to the next version of the operating system. And other companies don't have this. I mean, I'm not aware of any websites which print rumors about what's coming in the next Dell laptop. Yeah. So Apple likes to show something and have it be the first time you've ever seen it, right? Because they know you get a huge punch from showing something and you're like, wow, that's really cool. Like when Steve Jobs showed the first iPod, nobody had seen one. And people were like, wow, that is a really

cool thing, right? Whereas if it had been leaked and everyone kind of knew it was coming, then he shows it and you're like, oh yeah, that thing, I already knew about it. And so they work really hard to get that. And teams don't talk about what they're working on. You know, when you go to lunch with friends who work on different teams, you don't ask them, what are you doing? Because, you know, then they're just going to have to tell you like, I can't say, and that's impolite. So, right. You just talk about something else. Eventually David did figure out what they were building. You know, I talked to enough people and I realized like, okay, we're building a music player because, you know, people are worrying about, you know, how are we going to put music

files on it? And, you know, how are the headphones going to work? That's kind of a giveaway. Yeah. Yeah. So the iPod was announced, I think in the end of November in 2001. And it shipped a couple weeks later, you know, early December. So the timing was terrible. 9-11 had happened in September 2001. The US was in a recession between 9-11 and the dot-com bubble bursting. The US was in a recession. And Apple releases this $400 music player. Remember, this is 20 years ago. So this is probably like a $600 music player now based on inflation. And everyone's looking at this going, what are you guys thinking? And we didn't really sell many until the first quarter of 2002.

And we sold a couple hundred thousand of them, I think. And I remember thinking, I better start looking for another job because this project obviously is doomed. So this is why nobody would hire me for marketing because obviously I have no idea what's going to be popular and what's not because the iPod went on to be a huge success. But in the beginning, like nobody knew it was going to be a huge success. Marketing would come by, you know, every few weeks and they'd show us like a little come by, you know, every few weeks and they'd show us like a little clip of, you know, some basketball star was seen getting off the bus and he had the white earbuds and we're like, oh, cool. Some celebrity is using our products. You know,

isn't this great? Now, of course, you know, Apple sells a million iPhones a day. That's crazy. So I got, I think the third generation of the classic one. I remember it seemed expensive. I used it a lot. I found it very revolutionary at the time. But like in today's terms, it didn't like it just played music. But yeah, a lot came from this, right? Yeah, I think it did two key things for Apple. It showed people that Apple could do cool little devices.

And so people stopped thinking of Apple as the company that sells those weird computers that nobody uses except the guy in the graphics department. And people started buying little iPods and saying, oh, Apple makes cool stuff. I kind of like this, right? So it changed Apple's image. And it got Apple a lot of skill at making small miniaturized devices because that hardware team that made the iPod ended up doing a lot of work on the iPhone because they knew how to make tiny little devices. And Apple also learned how to write software for small embedded devices too. When the first iPhone came out, it was pretty hard to figure out how are we going to fit Unix? How are you going to fit that into the hardware that was in the original iPhone? It was quite a challenge.

I mean, there was some talk about whether we should just expand the iPod OS. And there was a bake-off, as they call it, where they did a couple different OSs. There was a little team of a couple of guys who got Linux running on early iPhone engineering. And they ended up picking the one that was based on the Mac OS. And that's what eventually turned into iOS that we use now. As the iPod sold more units, it started attracting more attention. And David's team's work started being directly evaluated by Steve Jobs. There was a period in the early 2000s when Apple was mainly known as the iPod company before the iPhone shipped. And Steve was intensely interested in the software, but of course, mostly

the user interface level of the software, right? I mean, he didn't care about the file system as long as it worked. It wasn't his interest. He wanted to know what are the apps look like on the screen. And I was in the group that wrote all those apps. My boss would have a weekly meeting with Steve and he'd go and show him what we had worked on. And usually he had questions like, you know, okay, there's several different ways we can go working on, you know, this photos app or this notes app, you know, what should we do? And sometimes Steve would say, you know, oh, you know, this is terrible. This is good? You know, what should we do? And sometimes Steve would say, you know, oh, you know, this is terrible. This is good. You know, you should do more of that.

But often Steve would do something totally different. I remember my boss would come back from his meeting with Steve. We'd all be huddled around his office waiting for him to come back to find out, you know, what did Steve say? Because this determined like the direction we were going in for the next week. He had gone in expecting Steve to say, like, did he like these screens we had done for the new player screen, you know, that showed you what song was playing. And instead, Steve had spent the entire meeting going into the preferences panel and just like tearing it all up and saying, I don't like this. I don't like that. I want you to rewrite all the preferences. And you know, the preferences had already been shipping for two years. We thought it was all settled. And my boss would come back and be like, well, guess what

we're doing this week? We're rewriting all the preferences. And I guess we'll see if we can get some closure on these other items next week. That's awesome. So that, that was actually a pretty common occurrence was Steve just going off on something completely unrelated because that's what caught his eye. The other thing was you did not want your demo to crash when Steve was playing with it. Remember, this is months before we shipped, right? And so we were building alpha builds of the operating system

and we would get like a half dozen iPods and each would be loaded with slightly different build to show off some different possibility of what we could do. And my boss would go into the meeting with all these different iPods. And there was kind of what we called the path he was supposed to go down, which is like what user interface items you are supposed to touch to show off. And you usually get, you know, like 30 seconds in before Steve would grab the iPod out of my boss's hand and would just start, you know, clicking around to play with it. And of course that's the worst thing ever because, you know, half of this

stuff isn't hooked up yet and Steve's going to hit something and then it's going to crash. And then Steve just decides that you're all a bunch of idiots because your software crashes. We hated it when that happened. Throughout these years of working on the iPod, David got to work on everything from device drivers to UI to secret government assignments. More on that very soon. But his favorite thing to work on was always the file system. There is one piece of software that I am still proud of, and it's something that converts the file systems between HFS and FAT.

When the iPod shipped originally, it only worked on Macs, and it used the HFS Plus file system, which was Apple's file system. And then at some point they said, you know, most of the world uses Windows. And so we want to be able to sell a Windows version too, and so we added support for FAT32 so that you could hook the iPod up to Windows. We shipped Mac iPods and Windows iPods, and you'd go into an Apple store and you'd say, I want a Mac iPod or a Windows iPod, and they had different SKUs, and the hardware was identical.

There was no difference. The difference was whether the disk was formatted, you know, for a Mac or for a Windows machine. And the marketing people came back and they said, you know, this is great because we're selling, you know, on both platforms. But we don't want to have to keep track of two SKUs when the hardware is identical. So can you make one which is going to work on either? Right. I said, that sounds simple. We'll just reformat the disc to be either fat or HFS, depending on, you know, what you hook it up to. And they said, oh, one more thing, though. We're going to do a marketing deal with some record companies and we're going to preload some songs onto the iPod.

They're going to pay us, and we're going to preload whatever they're promoting this season. And you can't erase that. I said, oh, this makes it a lot more complicated now. And so all the iPods would ship from the factory. I think they all shipped fat. And the first time you plugged it into a computer, if you plugged it into a Windows computer, it would say, that's great. It's on a Windows computer, it's just gonna stay fat. If you plugged it into a Mac, the first time you plugged it in, it would say,

oh, it's a Mac, I need to reformat to HFS. And so I wrote code, which would take the iPod, which is basically a computer, right? It's got a processor and a disk and all that. And while you were booted from the hard disk, it would reformat the hard disk from FAT to HFS Plus without losing any data. So all the files are maintained. And if you crash at any point or runs out of power

or someone does a hard reset at any point, you don't lose anything. I revert. So what I did is I would look for unused blocks in the FAT disk and I would write the HFS data structures there. And then at the final write was the write that changed the pointers at the beginning of the disk. So instead of pointing to the fat data structures, they pointed to the HFS data structures. And if it crashed during that one write, then you are screwed. But that's very unlikely, right? A single disk write of one block is very fast. So it goes from being a fat disk, which has a bunch of weird data structures that don't make any sense written in empty blocks, to being an HFS disk, which now has a bunch of weird data structures that don't make any sense written in empty blocks, to being an HFS disk, which now has a bunch of weird data structures, which don't make any sense written in unused blocks.

Guess what? The marketing department never actually closed the deal with the record companies. They never used this feature. I mean, they used the feature that reformatted the disk, right? So there's only one SKU. You'd buy an iPodod and the first time you'd plug it in, it would reformat itself as FAT or HFS, depending on what computer you plugged it into. But they never used the feature that I spent all that time on, which saved any data that was

already on the disc because they never did a deal with the record company for that. That's hilarious. I like your strategy though, right? It's at a good state the whole way through the conversion. Yes. I had to do that. I mean, one of the things I learned going from writing software, where if you were selling a few thousand copies a month, that was considered good sales, to working for Apple where you're selling millions, is that when you're selling millions and millions of units, every single possible thing that could go wrong is going to happen for someone. And so you have to make it absolutely bulletproof. If there's any way that something can fail, thousands of people

are going to hit that failure point. And that's not acceptable. David loved that working on the iPod meant that he could literally be full stack from bootloaders and device drivers to user interface. But working in a hardware organization also had its challenges. Apple has a separate hardware and software divisions. So most of the software engineers work for the software engineering division. But hardware obviously always has a little bit of software in it, right? There's always firmware chip or something. And working for a hardware division, like we were in iPod, means that the top people are all hardware engineers, right? The VP of iPod was

a hardware engineer, not a software engineer. And usually it's fine. It just means they don't know all the details of how we do our work, whereas they do know all the details of how the hardware engineers under them do their work because they are hardware engineers. But it also occasionally would cause problems. We were on these really tight schedules, working hard to get a new version of the iPod out every year. Apple shipped a new one for Christmas every year, just like they do now for phones. And if you work on a product where

you're going really fast, trying to get stuff out the door, you know, sometimes you don't always do the perfect design. Sometimes you kind of hack something together because it works, you know, and you think I'll come back later and I'll clean this up because, you know, this isn't great, but, you know, I can do a better job later. But right now, I have to get this done so we can ship. And, you know, after a couple years of that, you have code, which is pretty messed up and hard to support. Yeah. And that's when you want to refactor. And, you know, we'd go to our boss, and we'd say, you know, this code to support, you know, how menus are laid out, it's terrible, you know, this code to support, you know, how menus are laid out. It's terrible. You know, it has all these bugs. We should fix it. And my boss would say, okay, I'll

try and put some time into the schedule. And then it would go upstairs and, you know, the hardware engineers who actually ran the whole division, they would be like, what's this slack time? You've taken a couple of weeks out of the schedule to do this refactoring crap. You know, what's that? They thought we basically wanted to screw off and do nothing for a couple of weeks. And so they would cancel that. Right. And they would be like, no, no, we're going to add some more features that we can sell. Right. And we had a big blow up a couple years later where a lot of people, you know, we're having a lot of problems getting the code ready, getting it stable. Things weren't working. It coincided with a new development tool that we'd been promised not coming in on time. So that made us late. And the code was all just years of accumulated crap that we hadn't had time to clean up.

We ended up just working seven days a week for months getting it out the door. I know one engineer worked 34 days in a row without a day off. Oh, wow. And we'd also hired a bunch of people. So the team had grown. And because of the way Apple accounting worked, the budget for bonuses had been set at the beginning of the fiscal year. But by the end of the fiscal year, there were a bunch more people, but the budget for bonuses hadn't grown. So everyone got what we consider to be a small bonus for this super hard work. And at the end,

after we shipped the product, a third of the engineers quit. This was in the mid 2000s. And so that finally opened up management's eyes that, you know, you can't keep doing this because if you do it again and a third of the engineers quit next year, you're not going to have enough engineers to put on product again. Right. So that's when they finally realized that, okay, you guys aren't just making up this stuff about, you know, we want some time to refactor our code. You know, this is a real thing. This is something software engineers do

is they just, they take a couple of weeks off, they pick something that's really ugly and unmaintainable and they rewrite it to make it better. So we finally got some time to do that. In the midst of all these hectic schedules for yearly iPod releases, David's knowledge of file systems led him to get a very strange assignment. Yeah, so I was in my office one day and the director of iPod software, who was my boss's boss, came in and he said,

have a special project and I want you to work on it. There's some guys that are going to come over from the U.S. government and we're going to help them build some kind of device. And I said, what do you mean? And he said, oh, I don't know anything more than that. You know, you'll get a call tomorrow. And he said, your boss doesn't know anything about this. He just knows you're working on some special project. Don't talk to him. Only talk to me. Didn't that make you suspicious? Apple had a lot of special projects that would go on at various times. There were labs on the floor that were locked, you know, and my badge would not open them. In theory, you don't know what's going on. Sometimes you do know what's going on in them, but you're not supposed to,

so you don't talk about it. So my boss's boss says, you know, I want you to work on this secret project. And the next day, two guys show up, Paul and Matthew, and go into a conference room with them. And they said they're actually working for Bechtel, which is a large US defense contractor, right? They're not actual government employees. Bechtel has the contract from the government to do this. What did they look like? They were just regular engineers. They were probably about 30, just normal looking guys.

They were very good engineers. So I got them an office. I requisitioned an office. I got them a badge so they could come into the building. I told them what they needed in order to be able to compile and develop our code. So they needed a Windows machine. They needed to license the ARM development software from ARM. They needed to buy some iPods to mess with because Apple wasn't providing any material

support. Apple is just providing a place to do the work and they were providing me to help them. And Apple gave them a copy of the source code or I gave them a copy of the source code burned on a CD, but I told them you cannot take it out of the building. Once you compile it, you can take the object code on your working iPods out of the building, but you can't take the source code so like you tell them day one okay you're gonna need this machine and whatever and then what they come in with like a with a computer the next day and the shopping cart full of ipods they bought down at the apple store like pretty much yeah i mean i don't think it was the next day i think it took them a couple days but know, they requisitioned it, Bechtel bought it for them and they came back with this stuff and we set it up and building and learning your way around the

source code for iPod was kind of complex. And so my first job was just showing them, this is where things are in the source code. This is how to run the compiler, which was fairly complicated. We had a pretty deep build script. So once they got that going, then they said what they wanted. So they wanted to add some hardware to do something, which they wouldn't tell me what it was. They wanted a way to turn this on and off. How would they add hardware? So I never saw that part. I'm not a hardware engineer i was never in the office when they had any of that out so i don't know what it looked like or how they did it okay speculate they

pop open and like solder something in like i don't know what's the process or possibly they have something connected on the outside i don't know and then how do they wire that connected on the outside. I don't know. And then how do they wire that into the operating system? So that was the part that I helped with. They wanted a way to turn it on and off. So we picked one of the menus in preferences, which was really deep. One of the deepest menus you could go down. And we added a menu item,

which had some really boring, innocuous name, you know, like equalizer mode on off or something, something dumb. And I showed them how to add a menu item. I showed them, you know, how to detect, you know, when it had been selected. And then they wanted a place to store their data on the disk that wouldn't be detected. So I showed them how to add another partition. They wanted it to be completely innocuous. So you could plug it into a Mac or a PC. iTunes would pop up, would put new music on it, just like a real iPod.

And nothing strange would happen. iTunes would never say, hey, what's this extra thing? Right? And so we played with it a little bit until we found something totally innocuous that nobody noticed it was there. And then I showed them how in the operating system to get access to this partition and to write to it. And then they'd say, thank you very much. I was out of the office, they would close the door and they would get to work. How would they get the data off?

So it doesn't show up when you plug it into your computer. USB supports a bunch of different protocols. And MSC is one of the protocols for just hooking up a standard USB hard disk. If you just buy a cheap USB hard disk from Amazon and plug it into your laptop, that's the protocol it's running. And the iPod did that too. And the iPod also had some extra stuff we'd added into that protocol. You know, iTunes would send a special proprietary command. And if it got an answer, then it knew that it's talking to an iPod. And if it didn't

get an answer, it knew that, oh, this is just some random hard disk that was plugged in. Oh, so they could conceivably do the same thing. So there's something they're going to send and it's like knock three times and it's like, oh, here's your extra data. Yes, exactly. So presumably they had a tool that they wrote, which ran on their desktop computer, which would talk to the iPod and knew what to look for. Where in the government were they supposed to work from? So Bechtel had a contract with the Department of Energy.

And I know this because they gave me business cards and the business cards said they worked for Bechtel in a division that had a contract with the Department of Energy. And the U.S. Department of Energy manages the U.S. nuclear arsenal. arsenal so my guess is they were making some sort of secret geiger counter and they wanted to be able to take it around and look for i don't know people making dirty bombs maybe but not have it look like a geiger counter right someone's walking down the street with something that's clicking like you know a tv geiger counter you're like uh oh, what's going on here? So I looked it up. The Department of Energy in Nevada has the Nevada National Security Site. They have some role to play in making sure that nuclear weapons don't spread around the world. Yeah, I could see that. I mean, if you wanted to go and check if, you know, there's any signs of radiation in some place where, you know, you're not officially supposed to be checking.

You're on a tourist visa in Tehran or something. This would be a very useful device, right? If you're arrested with a Geiger counter, you're in trouble. But if someone looks at your iPod and the iPod works and it plays music and it's totally normal, you'd be much safer, right? Yeah, totally. So they built this and then... They built this thing. It took a couple of months and they were done and they left. And I never found out what happened to it. The interesting thing is, as far as I know,

Apple had no written agreement. This was all handshake under the table agreement. There were only four people at the company who knew about it. And as far as I know, we're the only people who knew that this happened. And there was nothing written down inside Apple, no email. And I assume because of like this, not talking about what's going on culture at Apple itself, did you keep the story to yourself for a long time? And you were like, by the way, I think I worked on a secret nuclear detection device or? Yes. Well, the other thing is if this thing is still in use, like you don't want to let,

you know, whoever our adversaries are know that, hey, this is one thing that we've done that, you know, you could look for to find out if there are any spies in your city. But of course, at this point, iPods are so obsolete, you would draw more attention to yourself trying to use an iPod now. If I want to verify this story, there's nobody who in fact can verify it. So actually, Tony Fidel, who was the vice president of iPod, when the story first was printed, Tony read it and he tweeted, yes, this really happened. It's quite possible there was some other Apple engineer helping them with the other parts that I never knew about, because I didn't have any reason to know about it.

I've included a link to Tony's versions of events on the episode page. He doesn't say a lot, but he does back up the story. I guess enough time has passed that people are now comfortable sharing the details. I think that's also the reason David's sharing about working on the original iPod. David actually loves the products that Apple builds. He's been a fan of Apple for a long time. One of his first computers was the Steve Wozniak-designed Apple II. I had a deal with a teacher. They had got a few Apple IIs

and they didn't have any software. And since I was known as kind of the kid at school who knew how to program, the teacher let me take one home for several months on the condition that I wrote a program that she wanted. And so it was a program to teach you basically how to balance a checkbook. And so that was how I earned the ability to have this Apple II at home for, I guess, a semester,

a year, something like that, was writing this software and giving it to the school. If you're going to teach somebody starting today how to program, do you get them an old Apple II? In a way, it's harder to learn now because the programs that you see are so much better. When I started to learn, GUIs didn't exist and everything was command line. And writing a command line program is simple, right? You just print stuff out and you read in what people type and there's no other interface. And so that's not very hard. And so it was easy for me to write programs that looked as good as the programs on

the computer I was using. And so, you know, in a couple of months, I thought, hey, I'm a programmer. When my kids wanted to learn to program when they were in high school, they're using these great video games on the Xbox that look amazing. You can't just sit down and write something like that, right? When I tried to show them how to write some simple software, they look at that and it's so far removed from the stuff that they see. It's like someone building a dollhouse compared to a real house. They can just see that it's not the same and they look at how much they're going to have to learn to get to write what they want. And it's discouraging. I'm on a mailing list with a bunch of engineers, and this is actually a common topic is, you

know, as people's kids get to a certain age, the kid is like, hey, dad, how do I write a program? I think JavaScript's good because you can do things to show up in the web browser. Most web browsers have a pretty good debugger in them. You can do some graphics. Yeah, for approachability, JavaScript's nice. It's good because you can do things to show up in the web browser. Most web browsers have a pretty good debugger in them. You can do some graphics. Yeah, for approachability, JavaScript's nice. It's right there. One thing that's different about your stories of software development,

from my experiences of software development, is just this kind of low-level working with hardware and trying to figure things out. Do you think that is missing from how people build software today? I remember reading somewhere that Leonardo da Vinci was probably the last human who could know all of human knowledge. Human knowledge has expanded so much that you could spend your entire life learning things and you would never come close to learning all the things that we have to know right now. And software is kind of expanded like that, too. When I started programming,

when I was a kid, right, there weren't that many languages, there weren't that many processors, graphics barely existed. You know, it was possible to learn most of what there was to know. In the iPod, you know, I knew most of the software that ran in it from the bottom to the top. The only thing I really didn't understand was the mathematics inside the audio codec, which converted MP3 files into audio waveforms. And now I just don't think anyone can do that. I mean, you have full stack developers, which means they know how to write an iPhone app or an Android app, and they know how to write some backend code that runs on a server, and they understand the protocols that go in between. But there's so much more

that they don't deal with. They probably don't know how the TCP IP stack works. They probably don't understand the memory management inside. They probably don't understand all the device drivers or the graphics package that puts everything on the screen. You know, on the back end, they probably don't understand, you know, the operating system that runs on the server. They don't understand, like, how to debug hardware with JTAG. I mean, it's great that there's so many more parts of software to get interested in. When I started, the web didn't exist, right? And now the web itself is such a large specialty. You can spend your entire career just working on

web stuff and never get into any other part of software. And it's enormous and complex, but it's completely different than writing a bootloader that knows how to talk to the registers in the hardware chips on the motherboard. They're changing the file system from big Indian to little Indian. With the podcast, I'm on a bit of a theme lately of like kind of going down the stack and you are definitely on that theme. Somebody said, hey, the benefit of digging into the details and getting low level is like the stack actually bottoms out. Like you actually hit hardware and you're at the end. In many other directions, you can just keep going forever. You can start to look into something and like never get back to where you started. Yeah, that's true. You know, at a certain point, you know, if you're at the bottom,

if you're talking to registers on chips that are on the motherboard. If you're looking at the assembly code that's being generated by the compiler, that is the hardware level. But on the other hand, there is stuff below that too. You talk to people at Intel, you know, who code the microcode inside the processor. They're even below the assembly language. Yeah, no, that's very true.

If I spend my life like writing Python code, but I'm a very dedicated software developer. I mean, do you think that my knowledge is too superficial? Am I missing something? in applying to work at the company. And we used to do a big part of the interview checking to see how well you understood memory management and pointers and stuff like that, because that was all kind of key information to see, are you going to be able to work in this environment? But as we moved into languages like Objective-C and then Swift, where the pointers are really hidden, I realized I shouldn't worry about whether people understand all the details. I have mixed feelings about that, right? Because then you work with people who can't read assembly language and you think, oh, they're missing

something. But on the other hand, they don't need to read assembly language because our tools have gotten so much better. So I have to admit it's a step forward. This is an interesting perspective. One thing I've learned from hearing all these varying opinions is that the context that you work in really matters. I'm not sure there's black or white answers. Game programmers value different things from embedded programmers who value different things from operating system writers, etc. So that was the show. You can find some more details on this, including Tony Fidel backing up the story linked to on the episode page, which you can find in the show notes in your podcast player. There's also a full transcript up with nice headings and links, and you can jump to different parts of the episode. I've been working hard to get professional

transcripts and nice episode pages up for all of the podcast, all the back episodes. And I'm finally done. A lot of people like to actually read podcast episodes, which I found a bit odd at first, but I've been coming around to it. So I tried to make these pages really nice and readable and shareable. And they have nice links so you can jump to certain sections. If you haven't seen what the new episode pages look like, take a look. They're also a great way to share an episode. But one more thing, if you or someone you know worked on some interesting and prominent or important software project in the past, and you want to share the story, spill the beans, let me know.

Email me at adam at co-recursive.com. Put story in the title there somewhere because I get a lot of strange email and that'll help me filter it. Until next time, thank you so much for listening.