Advent of Computing - Episode 136.5 - Data Center Disaster

Episode Date: August 4, 2024

I've gotten busy preparing for VCF West, so this time you get a short one! In this byte-sized episode we are looking at a short and strange story: that time a plane struck a software company, and the company turned around and used the crash in their own ads.

A light plane crashed through the roof of the Princeton Planning Corporation of America yesterday afternoon, touching off a fire that destroyed the two-story building owned by the Data Research Company. The pilot, Norman Jay of Philadelphia, led 13 employees of the Princeton Planning Company to safety. Jay, who was released from Princeton Hospital after treatment for contusions and abrasions of the forehead, claimed his craft, quote, lost power as he headed for a landing strip at nearby Princeton Airport on Route 206. The plane plunged halfway through the roof, missing two employees by less than 10 feet, according to William Sherman of Kendall Park,

who was in the building at the time. Sherman said most of the company's records were destroyed, but spokesmen for the data research company said the records were salvaged, although the building was destroyed by fire. End quote. From the Central New Jersey Home News, November 14th, 1969. Welcome to a bite-sized episode of Advent of Computing. I'm your host, Sean Haas, and as this episode's coming out, I'm probably driving back from VCF West. Long road trips are one of those unavoidable joys of rural living, and Advent of Computing's home office is about as far off the beaten track as you can get. Anyway, at VCF I gave a talk on edge-notch cards,

which was, assumedly, enjoyed by all. If all goes well, I should have a recording of that talk up on the feed pretty soon. Now, down to the brass tacks of the matter. I'm not the best at planning. VCF actually fell during one of my recording windows. I had to dedicate some time to getting ready for VCF and didn't really have time to do a full episode and prep for the conference. But never fear, I'm always one for hasty backup plans. So I present to you a bonus episode. If you're on my Patreon, then you're familiar with the format. Otherwise, this will be a taste of what you're missing out on. For these bonus episodes, I pick out an interesting topic that doesn't have too much meat on its bones.

This usually means there's a lack of sourcing or it's just a short story. Perfect as a quick taste of history. Today, I'm going to be sharing a short and rather strange story that I've been holding onto for a while. That's the story of the time that a plane crashed into the office of a company that sold backup software. The office caught on fire, had to be abandoned, and the company was able to spin the disaster into a nice ad for their software. So the perfect kind of fare for a bonus episode.

The company in question was Applied Data Research, or ADR. Most of the detail in this story actually comes from a conference talk given by one Philip Berg. He was a product manager at ADR. given by one Philip Berg. He was a product manager at ADR. If you're unfamiliar with that role, well, welcome to the software industry. Product managers are one of the many unsung heroes of software houses. Berg's talk is titled The Plane, spelled P-L-A-N-E, Facts About Data Center Accidents. This talk was part of a larger conference called Data Center Disasters from 1970. I only have a few pages of the proceedings, but it sounds like something that

would have been fascinating to attend. It's all about, well, disasters that happened at data centers and how those data centers recovered. ADR was one of the first software development companies, period, but they didn't start out that way. The company had been founded in 1959, well before commercial software was really a market. This was during the bundling era. It was customary for hardware manufacturers, like IBM, to bundle software with any hardware purchases. In that way, software was essentially free. At least, there wasn't a line item on your receipts for software costs. But a lack of a line item didn't mean that software was just growing on trees. It still had to be written. For larger companies like IBM, that was handled internally by their

own software development departments. Other companies relied on outside contractors. That's the niche that ADR fit into. They wrote software on a contract basis for computer companies like RCA and Sperry Rand. That would be the status quo for a while, at least. But things would change in 1964. This part of the story is told by Martin A. Goetz in an article on the Software History Center website. ADR had developed this automatic flowcharting program. It was, as Goetz says, speculative. It sounds like a proof of concept that they were passing around to their usual clients.

The problem was, no one showed interest. It sounds like a proof of concept that they were passing around to their usual clients. The problem was, no one showed interest. ADR believed that they had a winning program, it was something they wanted to use internally, but hardware manufacturers didn't agree with them. So, ADR decided to go to the open market, which wasn't quite a thing yet. ADR had to blaze their own trail and figure out how a third party could sell software direct to users. By doing so, there were many trials and tribulations, something I'll cover another time, I'm sure, but eventually ADR would make it. Their first program, this flowcharting package, was called Autoflow. By 1970, it was running on 1,200 computers, which,

for the time, is a very respectable install base. This success allowed them to expand to multiple offices and pursue other projects outside of pure contracting. The expansion also meant added complexity. That's kind of part of the battle that all software houses must fight. As you write more code and get more employees, you need to have some way to manage it all. To solve this problem, ADR pulled out the time-honored tradition of creating new tools. This is kind of one of those industry secrets. Developers actually spend a good deal of time writing software to help them write more software. It's just kind of how things turn out. First of all, they actually used Autoflow internally. This encouraged some better

programming practices in general. Autoflow generated flowcharts using annotations in your source code. Or put another way, it used comments. ADR pushed programmers to annotate their code, add proper comments, and put documentation directly into their source code commenting. They also implemented totally new tools just for their own use. One of those was called The Librarian. It's a wonderful name, I think. This was a very early version control program. Now, I don't want to get into all the details here, that's its own topic. But in short, Librarian managed changes in source code.

It kept track of changes made, could be used to roll back changes, and make regular backups of the current state of a program. This kind of software is invaluable to the programmer. When you're dealing with big projects, you have to have somebody to manage changes in your code. That's the only way you can really do collaboration on large programming projects. And ideally, that process needs to be automated. Librarian handled all of that for ADR. The other trick Librarian pulled was migrating code onto magnetic tape. Up to this point, most programmers had been using their ancestral medium, the punch card. Code was written onto cards, which were then loaded into the machine for further processing.

This worked, but it sucked, it was wasteful, and took up a whole lot of space. Librarian could manage code on magnetic tape, and could move code from punch cards onto that tape. This change gave ADR a lot more flexibility in how they programmed and how they dealt with their source code. Programs could be loaded and edited more quickly. Backups could be made much more easily. You know, the works. Tape was also just much more of a dense format. A single roll of tape could hold way more code than a stack of punch cards. That let ADR do wild things like, for instance, storing backups of their source code in safes or shipping backups off-site. That gets us to the main event. I mentioned that ADR had a number

of offices at this point. One, located in Princeton, New Jersey, happened to be in a particularly bad location. You see, it was across the street from an airport. The office no longer exists, but it was at Route 206 Center, Princeton. A quick check of Google Maps shows that that office would have been at the end of a runway. So, perhaps a disaster was an inevitability. Let us turn to data-centered disasters for the actual details of the story. To quote, At about two o'clock on the afternoon of November 13th, 1969, we heard the usual sound of a plane passing overhead. However, unlike all previous aircraft noises, this one did not gradually pass out of earshot. Instead,

it ended abruptly and unexpectedly. The impact itself was quite unspectacular. It was simply a dull thud. Thinking that a plane had crashed at the airport across the street, we rushed to the windows to see what had happened. It was then that we noticed that people were gathering in the adjacent parking lot, pointing and gesturing at our building. The plane had hit us. End quote. How's that for a fun way to prematurely end your work day? We can laugh a little here because, according to reports, no one was seriously injured in the crash. The pilot only suffered minor injuries, being treated and released from a nearby hospital on the same

day. No ADR employees were injured. But the crash was simply the beginning of the disaster. The plane hit the second story of the ADR office, and from there, a fire started. The fire destroyed the second story of the building completely. The first story was also badly damaged, so much so that the building was condemned. But somehow, not all was lost. The Princeton office housed a number of DEC computers and two IBM 360s. I don't know about the decks, but at least the 360s survived. They were on the first floor and, somehow, avoided complete destruction from both fire and water damage from the ensuing firefighting. Punch cards and paper were both ruined, though. Berg explains that it wasn't so much the fire itself that did

in the paper media, but rather smoke and water damage. The water is pretty self-explanatory, soggy paper is pretty useless, and soggy punch cards face a similar fate. Smoke damage, however, is also enough to destroy punch cards. There are very tight tolerances needed for a punch card to operate. If they warp from heat or smoke, then they can't be used. They're ruined. Despite the seeming devastation, ADR's Princeton office was actually back up and running in a matter of days. How is this possible?

Well, it all comes down to backups and contingency plans. IBM actually came in clutch here. Big Blue stepped in and helped move ADR's mainframes to a nearby warehouse. That let the Princeton office keep working. On its own, that was a huge advantage. There was also very little code lost during the fire. Berg states that 95% of their software survived. There were a number of factors at play here, all of which are tied into the librarian. The first is that ADR kept source code on magnetic tape. That actually really helped out here.

Magnetic tape is much more stable than punch cards, so tapes that were exposed to smoke and water were less damaged than, well, piles of paper. Tape offers a number of other advantages here. Remember how I said tape is much more data-dense than punch cards? That density allowed ADR to keep their source code backups in a fireproof safe. That safe wouldn't even have to be very large. With punch cards, that wouldn't have been possible, or at least not reasonable. ADR would have needed an entire bank vault to hold their code. But with tape, well, that's not a problem. their code. But with tape, well, that's not a problem. There's a third trick to this whole tape thing. Since code was so compact, ADR could afford to send copies of their backup tapes to other offices. Even if their safe had been compromised, there would have been off-site backups. That,

once again, wouldn't have been very practical with punch cards. Finally, there's one soft factor at play here. ADR enforced both code comments and documentation inside their source files. That meant that program documentation actually came out of source code. The very source code that was kept safe and versioned with the librarian. Flowcharts were also generated by encode comments using autoflow, so any flowcharts that were destroyed were recreated automatically using their source code that was preserved. All these factors meant that ADR was actually set up for success here, at least as much success as you can have after getting hit by an airplane.

This leads us to a pretty interesting spot. So, check this out. Berg actually points out that the crash had some benefits for ADR. To quote again from Data Center Disasters, One facet of this accident turned out to be quite amusing. The same fire which destroyed much important written material also managed to rid us of a large amount of obsolete material. This gave people the unique opportunity to actually start from scratch. All those listings and documents which should have been discarded months ago were suddenly gone. It was indeed a strange sight to see all those clean and clear desks, but what a price to pay for neat offices. End quote. Code and documents that weren't important

enough to be backed up, or old code that simply had fallen out of Librarian, was destroyed. That's a pretty funny way to get rid of outdated documents and idle projects that weren't going anywhere. The crash also gave ADR this wonderful marketing opportunity. I mean, how is this for some bona fides, right? The whole talk at Data Center Disaster was only possible thanks to, well, the disaster. well, the disaster. There were articles about ADR's miraculous survival in Computer World and Datamation, just to name a few trade journals and magazines. ADR went on to run full-page ads recounting the story of this plane crash. One, from 1970, in Datamation, opens with a big photo of the ADR office ablaze. From the ad copy,

quote, Librarian and Autoflow saved us inestimable time, money, and effort. We never used the term before, but both products served as vital insurance in continuing our normal operations. But possibly in your business, this aspect is not important. After all, things like accidents and fires only happen to the other guy. For a planned, peaceful demonstration, call or write. Applied data research.

End quote. How's that for turning a disaster into an opportunity? ADR's survival made for just about the best way to market backup software possible. Alright, thus ends our quick dive into the plain facts about data center accidents. Is there a lesson from this wild story? I mean, yeah, there kind of is. ADR did all these right things in preparation for a disaster. They had version control and backups in place. They had some really smart coding standards to make documentation easy. They even had off-site backups. They were ready for something to go wrong. For many offices, following Beck's practices is just an exercise. You rarely get hit by a plane.

But for ADR, the worst case scenario happened, and they won. I think it makes for a fun reminder of why it's worth preparing for the worst. Because for ADR, the worst actually happened. Thanks for listening to Advent of Computing. I know this episode's short, but I'm going to be back in two weeks with another full-length episode. It will either be a recording of my talk at VCF West or just a new episode. I'll decide on my drive home from Mountain View. Anyway, until then, thanks so much for listening, and as always, have a great rest of your day.