Advent of Computing - Episode 17 - The BBC Domesday Project

Episode Date: November 18, 2019

In 1086 William the Conqueror commissioned a survey of England that would come to be known as the Domesday Book. 900 years later the BBC would create a similar survey, called the Domesday Project. This new survey spanned two LaserDiscs holding over a gigabyte of data and 200,000 images, most of which were collected by students. It presets an amazing time capsule of the UK in 1986. Also contained within the disks were 3D virtual walks of the country side, and an entire computer generated gallery. So how did such strange technology come together to commemorate a 900 year old manuscript? Like the show? Then why not head over and support me on Patreon. Perks include early access to future episodes, and stickers: https://www.patreon.com/adventofcomputing Important dates in this episode: 1986: BBC Domesday Project Released

The Great Survey, The Book of Winchester, The Great Doomsday, or simply The Doomsday Book. The original text goes by many names. It was commissioned by King William the Conqueror in 1086, and it definitely sounds sinister. In reality, The Doomsday Book is much more benign than its name would suggest. It's actually a compilation of land and property surveys of parts of modern day England and some of Wales. The purpose of the survey was to account for the holdings of the British crown after the initial Norman conquest of the isle. The main goal was to work out how to charge taxes who would owe what to the king, but it also served

as a very convenient document to help with governing decisions. In modern times, the Doomsday Book is better known as a fantastic and very unique primary source. The text still exists. In fact, the whole thing has been digitized and it's relatively easy to find copies online. Over 900 years later, it serves as a fantastic time capsule of what England looked like in the distant past, and what the surveyors found important about the nation at the time.

So, why do I bring up an ancient manuscript on a computer history podcast? Well, the Doomsday book would lend its name to a much more recent project. In 1986, marking the 900th anniversary of the original, an updated edition would be released. And this new edition, the BBC Doomsday Project, would make use of all the newest tech of the day. Spread over two laser discs, you got a survey of the entire UK, 3D virtual walking tours of towns and cities, and even full motion video. Now, at least two of those things should sound a little bit too modern to be showing up in the mid-1980s. So, how did an ancient survey get reincarnated in the 80s?

And what exactly made the newer version so unique for the time. Welcome back to Adren of Computing. I'm your host, Sean Haas, and this is Episode 17, the BBC Doomsday Project. Today, we're going to be looking at a pretty strange story, and it goes along with a pretty strange piece of technology. The Doomsday Project is a lot of things. On one hand, it's an extension of the earlier BBC Computer Literacy Project. The creation of an updated survey would involve a lot of student-driven work. On the other hand, it's an amazing tech demo for augmented and virtual reality in the 1980s.

And on top of all that, the program was written in a now-dead computer language and distributed on Laserdiscs. This all makes up a really interesting combination of technology and intent that you don't really see very often. The original Doomsday Book was, for the time, a technical achievement in its own right. You gotta remember that in the 11th century in England, most people couldn't even read, so a huge set of volumes describing the entire country, well that's a pretty big feat. In that spirit, the modern day Doomsday project would need to be an equally stunning technical marvel.

And in the 80s, that meant virtual reality with the help of lasers. There are large portions of the British countryside that you can walk through, very similar to Google Street View today. Part of the menu system is actually modeled as an interactive 3D art gallery. And mixed in, you have statistics, pictures, and writings about Britain. And everything is controlled from the comfort of an 8-bit home computer. The whole thing really seems like a mishmash of tech demo and encyclopedia. The original Doomsday Book was, for the time, a technical achievement in its own right. You gotta remember that in the 11th century in England, most people

couldn't even read, so a huge set of volumes describing the entire country, well, that's a pretty big feat. In that spirit, the modern-day Doomsday Project would need to be an equally stunning technical marvel. And in the 80s, that meant virtual reality with the help of lasers. There are large portions of the British countryside that you can walk through, very similar to Google Street View today. Part of the menu system is actually modeled as an interactive 3D art gallery. And mixed in, you have statistics, pictures, and writings about Britain. And everything is controlled from the comfort of an 8-bit home computer. The whole thing really seems like a mishmash of tech demo and encyclopedia.

So let's take a look at the story of Doomsday, how the BBC partnered with tech companies to create a huge multimedia experience decades before anything similar would exist, and how exactly that leveraged school children to gather information. In the 1980s, the BBC was all about finding a way to democratize computing. The Computer Literacy Project, which started in 1982 and will continue on for the rest of the decade, was the BBC's first major push to try to get computers into classrooms, and by doing so, make computing just more accessible. As a result, Britain was quickly finding itself transformed into a country that was much more computerized. That was most visible in education.

Prior to the computer literacy project, very few schools had access to any sort of computers. But by the mid-80s, almost every classroom was outfitted. Outside of the classroom, workers were learning to adapt to a computerized future, and industry in England was adapting really well. This was the backdrop to the decade, and it would end up creating a lot of new opportunities, largely thanks to the work of the BBC and their partners. Around the same time the Literacy Project was being developed, Peter William Armstrong was just starting his career at the BBC and their partners. Around the same time the literacy project was being

developed, Peter William Armstrong was just starting his career at the BBC. First hired in 1971, Armstrong initially worked as a producer for religious programming on radio and eventually on television. From reading recollections of his co-workers during this time period, it sounds like he brought a really valuable insight to the table. Now, I know that faith-based broadcasting may not sound like the most interesting field, but Armstrong would bring about updates to how the BBC handled these kinds of programs. Some of these changes were technical, like putting a small studio mic in a preacher's robes to improve recording quality without distracting congregants. Other changes were more operational. Armstrong would lead a shift from simply recording a sermon

to creating a whole television program. They would include bumper tracks and interviews with the preacher and congregants. I bring up this early part of his career because Armstrong was an innovator, even at the very beginning of his BBC career. It's easy to see that he was willing to make changes and adapt to new technology, and that made his ideology fit right in with the computer literacy project, and in 1984 he'd find out for himself. Within the BBC, it was a well-known fact that the anniversary of the completion of the Doomsday Book was coming up, and there was already talks going on internally about how best to commemorate the event.

There was, of course, the easiest route, which was to just put together a documentary or maybe a series on the survey, but that's not really that interesting. Armstrong looked at this upcoming anniversary as a larger opportunity. His initial idea was to find some way to create an interactive documentary, but that quickly expanded in scope. Eventually, it would turn into the Doomsday Project. To quote from Armstrong, Rather than approach the subject from a historical point of view discussing how Britain has evolved over the last 900 years, End quote.

That sums up what's really at the core of the Doomsday Project. It would, from the beginning, have a dual purpose. It's, on one hand, creating an updated and usable catalog of the entire nation. On the other, a cultural snapshot. Something akin to a digitized time capsule. And from the very beginning, it was decided that Doomsday would have to be multimedia. It would combine still and moving pictures, text, sound, and numbers all into one finalized product. And all that data would be tied together with interactive menus, maps, and a searchable database.

In theory, that sounds like an instant home run. But some problems start to arise whenever you're looking at a project this large of scope. The planning stages of Doomsday would take place around 1984. That's the same year that the original Macintosh was released. Keep in mind that Apple's QuickTime wouldn't be developed for another 7 years. And most home computers, especially in the UK at this time, were 8-bit systems. There wasn't really much in the home market that could achieve what the BBC and Armstrong wanted. Even computers that did have graphical interfaces like the Mac wouldn't be

able to handle anything like interactive video at this point, at least not in very good resolution. The fact of the matter was that Armstrong's plans were just too modern for the state of mid-80s technology. This kind of conflict between idea and reality is something that appears often in the story of computing. It's a make-or-break kind of moment. You can either surrender, compromise, or just move on. And usually, it's some combination of all three. In the case of the Doomsday Project,

it's somewhere on the side of compromise. At the core of some of these compromises was the simple fact that it needed to be accessible, especially for schools. At the time, the BBC's computer of choice, and one of the more common computers in classrooms, was the BBC Micro. The Computer Literacy Project was centered around the BBC Micro, so the Doomsday Project would have to support it or at least something similar to it. Right away that imposes a lot of restrictions. The BBC Micro was an 8-bit computer, and it came with a maximum of 128 kilobytes of RAM.

That's not a powerhouse even by the standards of the day. The largest issue here comes down to the screen resolution. No matter how many tricks that you want to play, you can't display a realistic looking photo on an 8-bit computer. That's why the Laserdisc gets involved with this project. Now, Laserdisc is one of those strange, relatively short-lived formats. If you've seen one, it probably made an impression. Physically, they look a lot like a compact disc, but they're the size of a long-play vinyl record,

about 12 inches wide, and they're reflective on both sides. The first LaserDiscs were released by Philips and MCA way back in 1978. Initially, they were just used for home video. The format would die out before the end of the 90s, but for a time it looked like it was going to be the next big thing. And 1984 was right in that sweet spot. At the time, Laserdisc was the only format that could realistically hold large amounts of data. The CD-ROM didn't really exist yet outside of music, at least not in mass. So the BBC quickly selected Laserdisc as the storage media of choice for the project. Once planning and selection of technology was

out of the way, the BBC would start drafting up the technical details for the project. They knew that Laserdisc was their best bet, so they brought on Philips to develop the storage side of things. And Acorn, already partners in the computer literacy project, was selected to build the computer side of things. Despite the BBC's desire to create a simple and compatible system, plans would have to shift somewhat. The final design called for an upgrade to the BBC Micro, and a new controllable laser

disk drive built by Philips. The computer was called the BBC Master, and it was built around a slightly faster processor than the original Micro, and it came packed with more RAM. That side of the equation isn't super exciting, it's basically just an upgrade. The main purpose of the computer was really just to control the Laserdisc player, so as to read data and display information from the Laserdiscs. The more interesting part is Philips' custom Laserdisc Player. The machine was called the Philips VP415 Laser Vision Player, but I don't really like that name.

I prefer to think of it as the Doomsday Device, since it really is the core of the operation. Anyway, the device played custom formatted laser discs, later called LVROMs, and the principle of design behind these is really bizarre. So, a normal laser disc holds analog video information that's broken up into separate audio and video tracks. The newer LVROM replaces the audio part of the disc with a digital signal. A single side of one of these new discs can hold about 300MB of data, and with two sides per disk that works out to a little over 400 floppy disks worth of storage. And that's just on the data part.

The rest of the disk, where the video would normally be, was used to store small video clips and a collection of pictures. If you're paying attention, you might have spotted the weird part already. An LVROM is actually a combination of analog and digital data all on the same disk. That's not something you see almost ever. The data track is purely digital and the video track is still totally analog. The whole affair is just freaky to me. Now, another feature to keep in mind is that laser discs are totally seekable. They act as a random access device, so you can select exactly which data to read and which frame of video you want to display.

So that's where the traditional part of the hardware setup ends. You have a computer and you have some means of storage. Not really that out of the ordinary. But this is a case where the whole is greater than the sum of the parts. The Doomsday Device has one more trick hidden up its sleeve. Outside of the already strange analog digital laser discs, that is. Buried in its circuitry is what's known as an image compositor. That's a specialized component for overlaying multiple video signals on top of one another. This is really where the magic happens,

since that means the relatively high resolution pictures stored on the laser disc can be mixed with the lower resolution graphics and text that's rendered by the BBC Master computer. In practice, it worked something like this. The computer would request that the LaserDisc player skip to a specific image. The LaserDisc then pulls up and displays that on an attached monitor, so you have your nice analog higher resolution image. The BBC Master then sends out an overlay that has menus, labels, and probably some information

about the image to the LaserDisc player. That gets composited and superimposed on top of the analog image. And the whole option is handled in such a way that to the user, it just looks like you're pulling up a new file on the computer. In the absence of modern image formats and high-res graphics, it's a pretty effective trick, and honestly, it's pretty neat. That's the nuts and bolts. So now let's look at what was actually included in the Doomsday Survey. The Doomsday Project was broken up into two discs, the Community Disc and

the National Disc. The Community Disc was compiled to show a view of Britain by the British, so to speak. Most of the contents on the disc were produced by volunteers, making it almost akin to a pre-Wikipedia. Everything on the Community disk was browsable by map. Users would start up on a countrywide scale and then could zoom into smaller detailed views, and at every level there were pictures and writings about the region you were viewing. The community part of the disk was how the data was collected. The entire country was broken down into 4x3km rectangles, for a total of around 24,000 blocks of Britain. There was enough space on the disk so that each one of those chunks could have three full pictures and about 20 or so pages worth of text. This information was then collected by,

well, the community that's living in or around that specific 4x3 chunk of England. that's living in or around that specific 4x3 chunk of England. Collection was primarily done by schools, but volunteer groups were also involved. The overall program was pretty ingenious. With the survey being largely done by volunteers, it cost the BBC nothing to collect the data. And the hardware for this kind of data collection was already in most schools thanks to the Computer Literacy Project. for this kind of data collection was already in most schools thanks to the Computer Literacy Project. Students just had to write up descriptions of their hometowns on the BBC Micro that was already in their classroom. The text was then sent back to the BBC on either tape or floppy disk,

with the images sent as slides. By the time of release, over 9,000 schools would have sent in tapes. The entire process had a dual purpose. To quote from one of the 1986 release documents for the Doomsday Project, quote, The primary benefit of the Doomsday Project for schools will be the valuable educational experience of collecting the data. The long-term use and education of the finished disc is an important further benefit. End quote. The long-term use and education of the finished disk is an important further benefit. In this sense, the Doomsday Project would act as more than just a survey.

It's also an extension to the already existing Computer Literacy Project. Students involved would learn from helping the survey and continue to learn from the completed survey disks. and continue to learn from the completed survey discs. What makes the community disc such an interesting case is that it shows an almost unadulterated view of the UK by its residents. What was sent in for the discs wasn't edited. It was passed over to make sure that nothing illegal was being published, but outside of that, it was presented as a raw time capsule of the mid-1980s. I think it's a perfect corollary to

the project's namesake. The original Doomsday Book presented just what the Crown thought was important about Britain, while the 1986 rendition presented what British citizens found important about their home country. In an era before the internet really took off, this kind of mass by-the-people archiving is rare at best. The other half of the Doomsday Project was the aforementioned National Disk. If the Community Disk was the on-the-ground view of the country, then the National Disk is more of an aerial shot, so to speak. This disk was much more curated, mainly containing statistical and archival information. Part of this archive was ephemera, like scans of selected newspapers and magazine clippings. This came along with galleries of pictures and a handful of short

video clips of important events leading up to the early 80s. This was all accompanied by text descriptions of the entire subject material. That's all relatively mundane, really just a computer-controlled encyclopedia. However, once we stray out of that territory, we get into the more cutting-edge tech demo-like parts of Doomsday. And for me, as a bit of a stats nerd, the first stop has to be the census data. The Doomsday project came complete with the results of the 1981 UK census, augmented of course with a little bit of supplementary data. On its own, that's just a big ol' pile of numbers, not very useful. But thanks to some clever programming, it turns into something that's much more. One big piece to Doomsday was the software. It was more than just

a glorified controller and interface for a Laserdisc player. The entire census dataset was searchable and graphable. What's more, all the graphs were interactive, being built in real time. A user could easily mix around the axes and change what was being graphed with respect to what. And the final trick was overlays. Data that would pertain to a map could be plotted on the computer and then overlaid on the high-res maps that were stored on the Laserdisc's analog track. When you put that all together, you end up with a surprisingly robust data visualization package. Bundling up of the 8-bit digital graphs with the prettier analog images lends itself quite well to creating pretty and creative map overlays.

And honestly, it just looks pretty neat. and creative map overlays. And honestly, it just looks pretty neat. The next piece of the National Disc is where things really start to get wild. And that's the non-immersive virtual reality portion. That's right. In 1986, the BBC released virtual reality software on Laserdisc. You really gotta love this kind of stuff. Anyway, the VR component of Doomsday

consisted of trails called quote-unquote surrogate walks. These walks are analogous to something like Google Street View. You can navigate from point to point on a fixed path with views taken from actual images. You can even turn around 360 degrees and take in the scenery. And in some cases, you can walk up, go inside of buildings, and learn more information. Doomsday packed a total of nine of these virtual tours on the National Disc, with some walks using as many as 2,300 separate images to render the environment. Now, of course, this wasn't real 3D or VR, but rather, the computer just chose the next image to display based on how the user moved around.

The final tech demo-esque component was the Virtual National Gallery. That was a fully computer-rendered art gallery, and it served as the main menu for the National Disc. computer rendered art gallery, and it served as the main menu for the National Disc. Once loaded, users were able to freely walk around in the 3D rendered space. From the gallery, you could then walk up to, view, and maybe learn some more about the pictures that were hanging on the virtualized walls. You could also exit from the gallery doors and go into the aforementioned surrogate walks. Once again, the impressive 3D environment was mainly smoke and mirrors. The whole gallery had been pre-rendered on a more powerful mainframe computer, turned into separate images for each possible view a user could see, and then pressed onto the

disc. Just like with the surrogate walks, as the user moved around the gallery, the computer told the Laserdisc player to switch to the appropriate image. And I think it's really a great example of creative thinking. Of course, an 8-bit computer in the 1980s can't render a nice looking 3D world, but you can get pretty close by using some tricks. Now, all these sick tricks were accomplished thanks to some slick software. Most of the programming was done by Logica, a British IT company. Beyond creating the bulk of the software Doomsday shipped with, Logica also designed and rendered the 3D gallery in-house.

And just as a neat aside, the software was all written in BCPL, which is an ancestor of the much more popular C programming language. And something I find interesting here is that BCPL was never very popular, and on top of that, C would have already been available since it was released in 1972. It's just another mystery to think about why exactly Doomsday is written in a language that was already on its way out by the time it would be released. The entire project was completed in November of 1986. From conception to release took just about three years, maybe two if you're being modest. And I've seen estimates that say over a million people were involved with constructing the project, spread between volunteer data collectors,

employees at BBC, Acorn, Philips, and other contractors like Logica. While maybe not that large, the Doomsday Project was an impressive undertaking. The two master disks held a combined total of 1.2 gigabytes of data and over 200,000 images. Even by modern standards, that's a pretty impressive feat. So how was the Doomsday Project received once it got outside of the BBC? Once again, this is where things start to become a mixed bag. The entire package, which included the discs, computer, the laser disc player of course, and a trackball, came out to around 4,000 British pounds. Doing some calculations for inflation, that's a little over 16,000 pounds, or 12,000 US dollars. Originally, the project was planned to be accessible, but reality got in the way.

It wasn't really possible to make the system any cheaper, so right away it was out of reach to consumers and most of the schools. The British government did provide a subsidy for schools, but even with that, the system was still expensive. According to the BBC's records, only 1,000 copies of the Doomsday Project were ever sold. In the coming years, there were releases of other discs using the same custom Doomsday hardware, but there just wasn't enough of a market for them to be of any interest. Ultimately, cost wasn't the only issue. I think the September 1986 issue of Bits and Bytes magazine actually puts it quite well. To quote from Bits and Bytes,

The concept behind Doomsday is very far-reaching, since for the first time, large quantities of images and data can be held together. For publishers and knowledge workers, the media for communication will never be quite the same. And this really is just the start. Doomsday has one great limitation, in that it uses a fairly inflexible and centralized medium. Current discs have to be centrally mastered for reproduction, like an LP record, at some cost each time. A disc for use is published like a book, but this is going to change. Video discs that can write are on the way." The technology just wasn't quite there yet. The writable disc that was mentioned in that quote wouldn't be the magic bullet either.

The core problem was that a project this big really does need to be accessible and shareable for it to take off and have any hopes of recouping cost. The BBC identified this way back in 1983, but wasn't able to attain it. And who's to say that they even could have with the limits of technology in the 80s? Ultimately, Laserdisc was the only reasonable option, but going with that format ended up being a dead end. The LVROM disc format that Philips developed would only ever be used for a handful of discs outside of the Doomsday project.

And I think that one of the major reasons the technology behind Doomsday was so short lived comes down to bad timing. The CD-ROM was already well into development before the Doomsday project even started. By 1983, Sony and Philips would release the technical standard, called the Yellow Book, for that format. And by the middle of the 80s, CD-ROMs could store just under 700 megabytes of data. That's more than both sides of a laser disc. CD-ROMs were also cheaper to produce.

It's plain to see that the BBC ended up backing the wrong horse in this format race. If Doomsday had been produced maybe even one or two years later, they probably could have taken advantage of the CD-ROM format, and we could be having a totally different conversation today. The other issue caused by Laserdisc has only become apparent in the last couple of years, and that's a phenomenon called laser rot. The actual information on a laser disc is stored on a thin layer of aluminum, which is protected by another layer of plastic. Those layers are held together with an adhesive. Now, over time, that adhesive can start to fail

and let air get in. And if you've worked with aluminum before, then you know that it really likes to oxidize when it gets exposed to almost any amount of air. So effectively, every laser disc has a very real shelf life. Add to that the fact that the Doomsday Project only ever produced around a thousand sets of discs, and you can really see where this is starting to go. Not many copies are still around and readable. However, Doomsday won't slip quietly away. For such a short-lived product run, there's actually a lot of effort being put into preserving the Doomsday project, and I think that's a really good thing.

Some of these efforts are just keeping existing hardware running and taking care of disks so that they don't degrade any further. Other projects, like Doomsday 86, are working to preserve and recreate the entire Doomsday experience. Most of the data on Doomsday is already backed up on tapes and hard drives in the British National Archives. But projects like Doomsday 86 are working to use modern hardware to replace failing components of Doomsday systems, replace image disks, and eventually create a full emulator.

Now there isn't an easy way to experience Doomsday at home yet, but keep an eye out for that in the future. It looks like an emulator for the whole system might just be a few years away. Alright, I think it's that time again, so let's wrap this episode up. The Doomsday Project was ambitious. Perhaps too ambitious for the time it was created in. The goals were at once massive, but also so varied as to almost be at odds with itself.

But despite its large commercial flop, it actually accomplished everything it set out to do. In a time before the internet, the BBC was able to collect huge amounts of data from everyday people, compile it, and release it in a usable form. If estimates are to be believed, then the project brought together over a million Britons under a common goal. Data collection on the scale is something that we only rarely see today, even with the internet. Even outside of the data collection aspect, the foresight of its virtualized interface still shocks me. To think that 1980s technology could be used to generate interactive environments from real-world pictures, despite how rudimentary the approach is, still boggles my mind. And the fact that that was all somehow released together with thousands of pages of volunteer essays will always seem confounding but also amazing to me.

So with all its flaws, the Doomsday Project is a shining example of a time when ingenuity would outpace technology. Thanks for listening to Advent of Computing. I'll be back in two weeks time with another new episode. Until then, if you like the show, then why not take a second to share it with a friend? You can also rate and review on Apple Podcasts. The ratings really help the show get out there. If you have any comments or suggestions for a future topic, go ahead and shoot me a tweet. I'm at Advent of Comp on Twitter. And if you want to become a super fan, you can buy some merch from my TeePublic shop. I'll throw a link up in the description. And as always, have a great rest of your day.