Advent of Computing - Episode 104 - Sketchpad

Episode Date: March 19, 2023

We're finally taking a look at Sketchpad. This program was completed in 1963 as Ivan Sutherland's Ph.D. research. On the surface it looks like a very fancy drawing program. Under the hood it's hiding some impressive new programming techniques.   Selected Sources:   http://worrydream.com/refs/Sutherland-Sketchpad.pdf - Sutherland's Sketchpad thesis   https://www.youtube.com/watch?v=495nCzxM9PI - Sketchpad in action   https://www.computerhistory.org/collections/catalog/102738195 - Oral History transcripts

Hey, y'all want to hear about a cool source? I'm gonna just assume the answer is yes and keep going with this. In 2017, Ivan Sutherland donated a handful of documents to the Computer History Museum in Mountain View. Now, most museums, at least the ones on my radar, have archives attached to them. The same goes for most major universities. A lot of the materials in those archives arrive via this route. Someone who's done something of note will donate items to the

archive, or in their absence, the estate of that person will donate the items for them. This can introduce some sampling bias, but it's just kind of how things go. sampling bias, but it's just kind of how things go. Within Sutherland's donations were some strange looking papers. These papers had been printed at MIT in the very early 1960s. I'm not entirely sure the specific type of paper we're dealing with, but it's probably a little different than the usual. Think something along the lines of thermal paper. These prints came off one of Xerox's first computer printers, hooked up to an experimental computer deep in a lab on MIT's campus. As Sutherland would recall in a much later interview, it was a wonder that anything survived that machine. Quote, it had a big roll of paper which came through and came out of the Xerox

printer into a steel bin, which had a handy cover to put over it in case there was a fire, because the fuser would sometimes get too hot and the paper would come out on fire. Well, you had to cover it up, end quote. But hey, this is experimental technology, right? Connected up to, well, some more experimental technology. And besides, it worked most of the time, as long as it wasn't starting a fire. Something neat about this printer is that it used a modified CRT to create its final impression. This allowed for some pretty fancy typefacing. create its final impression. This allowed for some pretty fancy typefacing. The documents in question have superscripts and subscripts. Common today, but at the time, that was pretty slick. These documents are important, on one hand, because they're some of the first pages ever

printed on a Xerox printer. That, just on its own, is pretty neat. But the contents, well, that's the real gem. The documents are a printout of source code for a program called Sketchpad. Preserved on these sheets, we can see the first object-oriented program. At least, so it's been claimed. Welcome back to Advent of Computing. I'm your host, Sean Haas, and this is episode 104, Sketchpad. Today, we're examining something that I really should have probably covered a long time ago. I guess I feel that way about a lot of topics. We're going to be looking at Ivan Sutherland's Sketchpad. It's a fascinating program with some wide-reaching implications. On the surface, Sketchpad seems pretty simple. It's just a neat graphical CAD program, right? You draw shapes,

connect them up to other shapes, do fancy transforms on lines, you can even import drawings into your bigger drawing. Really basic computer-aided drafting stuff. Well, here's the cool part. Sketchpad was developed in 1962 for Sutherland's PhD thesis. So we're dealing with a really early example of a graphical user interface. The entire thing was controlled via a light pin and a CRT. You actually draw directly on the screen. For the time, that was a really big deal. This means that Sketchpad predates other GUI developments like Engelbart's NLS. It very handily predates graphical work done at Xerox. So the big question here is, where does Sketchpad fall in the larger context?

Was Sketchpad inspired by earlier works? And by the same token, what later works did Sketchpad inspire? Both of those questions get tricky. I've seen a few articles that try and shoehorn Sketchpad into the larger hypertext lineage. I think this is a really easy assumption to make. Sutherland's thesis falls between Bush's As We May Think and Engelbart's NLS. So the argument goes that Bush inspired Sutherland, who in turn inspired Engelbart. Now, I have my reservations about this viewpoint. First of all, in a 2017 oral history interview, Sutherland claims he can't remember reading As We May Think.

Sutherland's thesis includes no citations to Bush's article. Augmenting human intellect, the big manifesto that Engelbart wrote that led to NLS, well, it mentions Sutherland, but only in a very offhanded manner. It just says that Sutherland is also working on a GUI-like system over at MIT. My cursory examination points to a few connections, but I want to take a deeper look at things. With my horde of sources and notes, I may be able to solve this for myself. So this episode is going to be about Sketchpad, its development, and its connections to the larger world. No matter what, Sketchpad was a seminal work. Sutherland was one of the first people to present a graphical user interface. But is this secretly a hypertext episode?

Well, we'll have to find out. But before that, I need to slide a quick plug in. It's for Notes on Computer History, which is getting really close to being finalized. If you haven't been keeping up, Notes on Computer History is my attempt to create a non-academic journal for talking about, well, the history of computers. The goal is to provide a space for anyone interested in the topic to write about it, and you don't have to have any prior experience. Right now, I'm looking for two more articles before I finalize our first issue.

It's been a bit of a slog, but the end is really closing in quick. So, if you're interested in the history of computing, which I assume you are if you've made it this far, then please reach out to me and submit a draft. You can find information on how to submit at history.computer. With that out of the way, let's get into the episode. Ivan Sutherland is a really interesting dude. Usually when I do episodes that are heavily focused on a single person,

I start by turning to an oral history or some other long-form interview with the subject. I start by turning to an oral history or some other long-form interview with the subject. Turning through those transcripts gives me a quick, cursory understanding of the individual. Then I can dive into more old sources, get a little more primary. In the case of Sutherland, nearly every page of his oral histories include the line, this is in brackets, laughing. It's an indication that Sutherland or one of the interviewers punctuated a phrase by, well, you know, laughing. I don't see that a lot,

and frankly, it goes a long way towards giving me a positive impression of someone. Sutherland grew up in a particularly advantageous position. He was born in 1938, which puts his younger years smack in the middle of the development of the digital computer. He was 7 when ENIAC first sparked to life, 14 when the first compiler was developed in the bowels of UNIVAC. This was all still cutting-edge stuff, but Sutherland was born into the first generation of actual computer scientists. By the time he entered college, computers were already research tools. There were machines that could actually be used, not just machines waiting to be built. He was in a position to enter a totally new field. In 1955, Sutherland enrolled in the electrical engineering program at the Carnegie

Institute of Technology. By that time, he was already a programmer, if you can believe it. Nowadays, it's kind of common for teenagers to get into programming. It's one of the many ways I idled away my personal misspent youth. But in 1950, well, that just wasn't common. Was Sutherland some kind of boy genius? Perhaps, but he also happened to have good connections. Both of Sutherland's parents were academics in some capacity. As such, they knew other nerds. It's kind of how things go. One of these family friends was Edmund Berkeley, a co-founder of the Association of Computing Machinery. He had taken an interest in the education of Ivan and his brother Bert. At one point, most likely in 52 or 53, Berkeley would

send the two boys home with a computer. That machine was called Simon. It's a two-bit relay machine of Berkeley's own design. Not exactly cutting-edge technology, but the idea of two teenage boys borrowing a whole computer in 1953, well, that kind of boggles the mind. The fact that the 2 would program for this machine, well, that's just another level of amazement. Ivan would develop a division program for the 2-bit machine. That may sound a little simple by today's standards, but Simon wasn't the most adept computer. To quote from Sutherland, division by 1 was straightforward. It took the numerator and put it out. Division by two is not hard. You shift and put it out. Oh, division by zero was half an inch of tape.

Division by one was another inch of tape. Division by two was three inches of tape. The rest of the six feet of tape was division by three. End quote. I want to recount the story because I think it gives us an interesting look at Sutherland's background. He was given unique opportunities in his youth that wouldn't have been possible for the previous generation. He was born at the right time to stand on the shoulders of giants, so to speak. Sutherland graduated from CIT in 1959, then moved over to Caltech for his master's, which was completed in 1960. After Caltech,

he enrolled at MIT for his doctorate, once again in electrical engineering. It's his time at MIT that will be of main concern this episode. Shortly after joining campus, Sutherland would get a new job. One of the most storied institutions at MIT in this period was Lincoln Lab. During the 1950s, they had provided major contributions to Project SAGE. They had built TX0, the first transistorized computer. And as Sutherland entered the lab, there was a new machine on the block, TX-2. Now, despite the fancy name, TX-2 was actually the direct successor to TX-0. There was, at one time, a TX-1 planned, but it didn't really get off the ground. And just as a fun aside here, something I never realized is that TX0 encased its transistors in these little glass tubes. Superficially, its circuits actually look

just like slightly smaller vacuum tube circuits. That's neither here nor there, I just think it's neat and why to squeeze it in. Anyway, TX2 represented a relatively new type of computer. Lincoln Lab had already completed the main thrust for their work on Sage. That meant that the outfit now had a good deal of user interface experience under their belt. Plus, there's a lot of new technology floating around the lab. For instance, the light gun. One of the first digital pointing devices had been developed at MIT's Servomechanisms Lab, another group on campus that contributed to SAGE. If you wanted to be in a position to flex some new tech on a new computer, well, Lincoln Lab would be a pretty good option in this period. It didn't hurt that they also paid a salary.

TX2 was special in a number of ways. Its primary designer, Wesley Clark, wanted this new machine to be a testbed for new ideas in human-computer interfaces. That's the fun catch-all for everything from GUIs to text interfaces to really anything that plugs a computer into a machine. As such, TX2 was a mix of some new and some old ideas backed up with some real horsepower, frankly. The machine started life with 64,000 words of magnetic core memory. For the time, that was an almost ridiculous amount of space. One popular contemporary machine, the IBM 709, that only shipped with half of that memory. Of course, there are caveats about byte and word size differences between 1958 and today, but

no matter how you cut it, TX2 had a lot of RAM. Some of those sweet magnetic core planes had even been stolen from TX0 itself. The biggie here, the cool feature that would set this new machine apart, was one of its output devices. TX2 had an integrated CRT display. had an integrated CRT display. This is some more tech that was ported over from Whirlwind, TX0, and work related to Sage. Along with the CRT came a light pin. Now, this is technology that gets mentioned on the show occasionally, but I don't think I've ever properly explained it. The first light pin, really something more like a light gun, was developed for Whirlwind. It allowed an operator to point to a location on a CRT and pull the trigger. To an operator, it just seems like you're pointing and selecting a pixel on screen.

It's a really simple type of intuitive input device. If you've ever used an NES zapper, then you can probably attest to the light pin's simplicity. That device uses the same technology. So how exactly does it work? How does the pin know where it's pointed? This is one of those neat smoke and mirror tricks. Inside the light pin is a photocell, a small light sensor. It's basically just a chunk of silicon that can tell if it's light or dark out. Add a button or a trigger and, well, that's actually just about the whole hardware side. The light pin itself is remarkably

simple. The most complex piece is probably a little lens they put on the front of it. simple. The most complex piece is probably a little lens they put on the front of it. The tricky part is all in how the computer drives its CRT. These old displays work by scanning an electron beam across a thin layer of phosphor on the inside of the screen. I say scanning, but that's a little bit inaccurate. There's this thing called a magnetic yoke that can steer the electron beam to any position on the screen. It has x and y controls over the beam. In normal operation, this yoke will attempt to steer the beam in a pattern of horizontal lines. The electron beam isn't always on, so most of the time the yoke is just churning its magnetic field around. By carefully timing when you pulse the electron beam on and off, you can draw fun little pictures

on the screen. That's how everything from a TV to this specific CRT functions. That fancy timing is handled by the computer in this case, or at least its graphics hardware. This means that at any given time, there's some part of the machine that knows where the yoke is pointing, and if the electron beam is on, also. CRT displays have pretty high refresh rates. That means that the entire screen is swept with pretty high frequency. Call it 60 times a second or so. That's fast enough that the human eye won't notice a few strange frames. So you have some wiggle room. To get the light pin to work, the computer just drives the electron beam hard for an entire frame,

which fills out the entire surface of the CRT with light. Of course, this goes one chunk at a time. Now if you hold the photocell in the light pin up to the screen, it will detect when that sweep comes up to the indicated point. Do a little math, process some signals, and the computer can calculate where on the screen that pin is pointing. That's the brute force method. There are, of course, much smarter ways to handle light pin detection.

It all comes down to trying to detect blips of light and then syncing that up with what's drawn on the screen at some given time. The light pin, or gun, has seen a number of uses throughout the years. Its first big coming out was, of course, Sage. That's what it was designed for. In general, we can think of the light pin as an analog to the mouse. In fact, Engelbart had considered the light pin as a possible pointing device during the course of his research. The problem, he realized, came down to ergonomics. Direct two-screen pointing devices are really intuitive to use, but a long session will leave your arm fatigued. The trick is to use this technology sparingly and smartly. In the case of Sage, the light gun was only part of its

interface. It was augmented with a pile of buttons and blinking lights. Sutherland would employ TX2's light pin in a similar manner. So how do we bridge this gap? How does Sutherland go from getting a job at Lincoln Lab to developing his graphics package? The program doesn't just appear from nothing, after all. That would be pretty boring. TX2 had been, in a sense, designed for human interface research. Digital drawings were showing up on the machine pretty early on. In that same oral history interview that I cited earlier, the one with the laughing, Sutherland recalls that the first graphics program for the machine was developed by one Hirsch Loomis. This would have taken place probably in 1960. The initial program was crude.

TX2's display was mounted over a set of dials that were intended to serve as general inputs. Loomis simply tied into those dials to drive a pixel around the screen. This should be a familiar setup. You had one dial for X movement, one for Y. By programming the pixel to leave a trail, a user could draw out simple images. It was a neat demo, it may have been copyright infringement, but not very impressive to Sutherland. That said, graphics were on his mind. Part of getting a PhD is, well, doing a thesis. You have to do some sort of original research,

create some new contribution, and then be judged on that work. By the time Sutherland was working at Lincoln Lab, he had the idea for this project. It would play into the growing interest around graphics at MIT. It seemed that even Wesley Clark, TX2's designer, was ready for this type of project to commence. To quote, was ready for this type of project to commence. To quote, I had seen the TX-2 the summer before, and I knew it had the CRT on it. When I went to Wes and said, I want to use it, I said, I want to use it for engineering drawings. I mean, I was quite upfront about that. I had had the idea that it could do that earlier, and so I proposed that to Wes, and he said yes. And years later, There are plenty of reasons that Clark would have been waiting for a Sketchpad-like program to be developed. I think a lot of researchers and

engineers, whether they knew it or not, were waiting for something like Sketchpad. Up to this point, drafting had been an almost fully analog process. We're talking pencil and paper kind of stuff. Your most advanced tools were along the lines of the ruler, or perhaps the stencil, maybe a desktop calculator if you were really crushing some numbers. But the drawing, the actual process of devoting designs to paper, was almost impossible to automate. Strides were being taken towards a solution here. The larger field was called Computer-Aided Design, or CAD. At this point in time, CAD was really only tackling one aspect of this problem.

Over in another lab at MIT, the newly named Computer Aided Design Group, there was this tool called APT. This is another advent of computer alums. See episode 49 for a look into this one. Anyway, APT was a programming language designed for controlling CNC machines, basically automated digital mills. APT itself takes an interesting approach to this problem. You see, it's not some kind of fancy drafting program. It's a fully-fledged language. At least, it's kind of a programming language. Instead of being general purpose with conditionals and all that jazz, APT is used specifically for specifying geometry and tool paths. You define your geometries using basic components like lines, points, and circles. From that small set of options, you build up objects.

These objects are treated like variables. They have names, they can be reused. You can build more complex objects by combining multiple smaller objects. Once you have all your geometry set up, you then define a toolpath. During that path part, you get to reference all your fancy points and lines. APT itself provides tools for checking the paths make sense with your CNC machine. The cool part here is that everything is parametric, and a change to a parameter carries over to the rest of the APT program. The canonical demo for APT is this triangular metal ashtray. Don't ask why, that's just how it is. The ashtray is a big rounded triangle with a middle divot for holding

ash and two smaller divots on its side for holding cigarettes plus some text lower down. If you wanted to scale that up from, say, cigarettes to mega-cigarettes, then you just need to change a scale parameter. Perhaps a cooler example would be changing the shape of the cigarette divots. There are two of these divots that, at least in theory, are just references to some divot variable. Instead of going into the code and changing each divot individually, APT lets you make a change in a single place. All you need to do is change the initial definition of the divot geometry, and that change will propagate to those two instances of the shape. The final part that made APT supremely useful was its ability to visualize designs.

The CNC nerds had an oscilloscope that was wired up to the computer that ran APT. With a little bit of code, it was possible to show tool paths on that display. Mind you, this is pre-TX2 stuff. mind you this is pre-tx2 stuff apt wasn't intended as a tool for drawing fancy graphics but it could do that if you wanted these images were used more for debugging and testing basically development tools while working out a machine path it's useful but not the end goal. With this, we can start to form Sketchpad's lineage. There were graphic demos developed for TX2, that's one plank in the platform. Sutherland saw that firsthand, so we have a solid connection. There was also an existing program at MIT for managing a CNC machine that happened to spit out engineering drawings while it was cutting metal. Was there a connection between Sutherland and APT? The answer is an unequivocal yes. Our

evidence actually comes from Sutherland's thesis on Sketchpad, which is a pretty neat source. You see, he has a history section in his thesis. Call that a little self-aggrandizement or call it good record-keeping. Either way, I like it. Sutherland explains that one of the prior arts he looked at while preparing Sketchpad was APT. Moreover, Sutherland thanks Douglas Ross in his thesis. Ross was the developer of APT. I believe this helps us answer one of my primary questions. Sketchpad traces its lineage back to this mix of computer interface research and computer-aided design tools. Add in Sutherland's engineering background and I think we're set. These early sources make no mention of Vannevar Bush, as we may think, or the Mimics. No chains of ideas either. So why have others made this connection? Why can we read articles that say the Mimics inspired Sutherland to create Sketchpad?

Well, I offer you two hypotheses here. The first is that there was, in fact, a tenuous connection between Bush and Sutherland. That connection was named Claude Shannon, who was Sutherland's thesis advisor. He had also been a friend of Sutherland's family since Ivan was a child. Back before the creation of digital computers in the interwar years, Shannon worked under Bush at MIT. So, Sutherland is actually just one step away from Bush. However, Shannon didn't have anything to do with the MIMEX. Shannon worked with Bush's differential analyzer, an analog computer that was built in the 1930s. Shannon wasn't around when As We May Think was written.

So while there is a link, I don't know if it's a meaningful one. Theory two is that the connection was made ex post facto by third parties. Then that connection was either misinterpreted or spread. I think the two most likely vectors here are Alan Kay and Ted Nelson. Kay and Nelson were both involved with the user interface revolution that led to the modern GUI. Kay was more on the graphics side, while Nelson was slanted towards hypermedia. That said, there is some crossover, especially ideologically. Both researchers have spoken glowingly about Vannevar Bush and Ivan Sutherland and even Doug Engelbart. Now,

I haven't seen some smoking gun here. That said, we do have some connections. There's this 1986 lecture that Kay gave at a conference called The History of the Personal Workstation. In this lecture, he cites Sutherland as one of the influences that led to his ideas on user interfaces. He also cites Vannevar Bush. The lecture doesn't make an explicit connection, but it puts the mimics and sketchpad on the same timeline that leads to the GUI.

I think we can see how that could be misinterpreted as an implicit connection. As for the Nelson angle, well, that's a little more difficult to quantify. The corpus of his work is both vast and in some parts inscrutable. I know he discusses both Bush and Sutherland in Computer Lib slash Dream Machines. That's his first and most interesting book. That said, I haven't found any places where Nelson links the two ideas together in the same breadth. I think that's probably because Nelson saw Sketchpad more as a graphical application than a hypermedia one. This is all to say that I don't think Sutherland had the mimics in mind when he was working on his thesis. Sure, he may have heard

of Vannevar Bush, but when asked about As We May Think in his oral interview at the Computer History Museum, Sutherland said he had no memory of the paper. I'd say that's pretty definitive on its face. We have the background leading up to Sketchpad pretty well covered here, so I think we can safely turn to the program itself. The software would evolve somewhat after its initial creation, so just for reference, I'm sticking to the implementation described in Sutherland's thesis. And fear not, the document leaves very little to the imagination. On the most basic level, Sketchpad is a vector drawing program. That should serve as a good surface-level description. The interface was really simple. It's just a CRT, a light pin, and some buttons and dials.

Each button corresponds to a different action, with the light pin only being used to select and draw vectors. All you had to do was sit down, hit the button to draw a line, and then use the light pin to specify a starting and an ending point. Repeat that until you have the shape you want. While some of that will sound familiar, there's still something deeply archaic about this interface. Sketchpad is cited by many, including people like Kay and Nelson, as one of the earliest examples of a graphical user interface. We must be very careful not to misinterpret this. It's important for us to understand that this doesn't mean Sketchpad looked anything like a GUI you or I would use today. The buttons here aren't little boxes on the display. They're physical push

buttons on TX2's control panel. That is radically different from modern ideas of user interfaces. This is software tailored to very specific hardware, an interface backed by a very specific physical form factor. Sutherland didn't somehow create all of the magic that makes a GUI work, but he did hit upon some of that magic. The most clear point here is, well, the ability to point. We can't say who was the first person to point to something on a computer. Looking for that would be an exercise in frustration, much like my recent dalliance with text interfaces. Pointing to select something is kind of just human nature. How do you indicate you want something? Well, you point at it. It's

an unambiguous gesture, so I think it's obvious that it would be adapted for digital use. As I've explained, the light gun was basically developed for use in the SAGE project. It's a tool for pointing at something on a radar screen, and that action of pointing is then fed into some software for analysis. While we can definitely say that Sutherland didn't invent the idea of pointing at a computer screen, he did use it to great effect. Pointing was used to select points for drawing vectors, and it was also used for selecting existing vectors. This is simple stuff, but it adds a natural feel to Sketchpad. In APT, for instance, a programmer had to specify points via numeric coordinates. In Sketchpad, you didn't even have displayed axes, just a blank screen to draw on.

There's another little complication to the light pin here. The flick. During some operations, such as drawing closed shapes, it was important to have a way to tell Sketchpad you were done selecting points. This could have been done via another button press, but Sutherland went an interesting route here. When you're finished making selections, you flick the light pin off the display. Now, I was a little confused when I first read this. I didn't know if Sutherland meant it as a literal flick of the wrist or was talking figuratively.

You know, with the flick of a pin, the image is complete. Well, dear listener, I can report that the flick is very real. There is extant footage of Sutherland using Sketchpad. I'll include a link to a clip if you want to see the flick yourself. And it's exactly what it sounds like. It's this little dramatic twist of the wrist to complete an action. And I really like this feature. Partly because it's fun, and partly because it's fun and partly because it's just practical. Just think about this for a minute. How would you signify the end of an operation? You could have the user press another button on the console. That would work, and you could even say it's consistent UX or something. However, that breaks the user out of CRT land.

You have to leave your workflow to look at the console, find the button, and hit it. You could try and make that less of an issue by putting the start and stop buttons close together, or maybe using the same button for both operations, but you're just trying to dress up a bit of an issue. The flick, well, I'd argue that's pretty intuitive. Done? Do a little flourish. More than that, it keeps the user glued to the CRT. It helps you stay in the workflow of drawing vectors. If you're trying to draw some complex geometry, you aren't going to make one closed shape then call it a day. More likely,

you're going to be building up a lot of different polygons. The flick means you can stay in front of the screen and draw line after line without moving your view. You might have to have one hand on the start button, but you're still plugged in to the CRT. Maybe that's reading too much into this tiny feature. The bottom line is, I think the flick was a really good decision. Another really important feature from Sketchpad that's obviously connected to the modern day is how it handles magnification. TX2's CRT was a pretty small display, all things considered. If that was your only space to draw in, well, you wouldn't be producing very big sketches. SellerLens' solution was to allow the user to draw on a much larger virtual canvas that you could zoom in and out of. You were able to view the

entire canvas, or just a magnified section of it. This is one of those neat solutions to a specific hardware limitation. Sure, you'll never get an infinitely large display, but screens in the early 60s were particularly small. Of course, this does introduce some complexity to Sketchpad's code. The actual drawing exists in this virtual space in memory somewhere. Chunks of that space are read and twisted around to make the portion displayed on the CRT. In theory, that virtual space could be any size, and it has to be accounted for as you make modifications to that small visual portion of things. Moving an object on the CRT, well,

that might change things in this ethereal realm of memory. The fact that it all works really, I think, puts a feather in Sutherland's cap. We can look at this magnification feature a slightly different way. Sutherland uses the word window to describe this feature. At least, he does in a handful of places in his thesis, as in, quote, the magnification feature of the scope window into the page makes it possible to draw the fine details of a drawing. The CRT isn't the page, it's only a window into a larger space. At any one time, you're viewing part of the sketch, not all of it. You can move that window around to take a look at different parts. Sutherland uses the term window as a way to explain the magnification feature. At this stage, it's kind of a metaphor, but the wording actually sticks around. I don't think this really needs

much explanation, even. Windows are still a core feature of modern GUIs. Now, I'm not entirely sure how the word gets adopted after Sketchpad. That's a discussion for another episode. But it's clear to see how Sutherland's tiny TX2 windows are, in a lot of ways, similar to windows today. Right now, I'm editing a text document in a window. The window is only the size of my screen, but the document itself is much larger. I can scroll around to browse the full text. In effect, I'm moving the window over my document. It's just an effective metaphor, plain and simple.

over my document. It's just an effective metaphor, plain and simple. Another important feature of Sketchpad, and what really elevates it beyond a simple drawing program, is the constraint. This is what gives the program some real depth, and starts taking us in a more technical direction. Constraints allow a user to impose, well, surprise surprise, constraints on certain vectors. These are rules that TX2 applies to the lines you've drawn. This is a little hard to talk about in the abstract, so let me switch to an example. Let's say you want to draw a perfect square. All lines equal all corners 90 degrees.

If you were doing this with pen and paper, you'd probably start with a ruler. First, you mark out your points, measure to make them all equidistant, then you draw straight lines using the edge of your ruler and a pen. All in all, it's a bit of an annoying process. In Sketchpad, you have a much easier way around the hassle. process. In Sketchpad, you have a much easier way around the hassle. You simply draw a shape with four points, flicking the pen when you're done. The next step, and the final step, is to apply a constraint. To get a square, you just tell Sketchpad to make each line equal in length. Then you get to kick back and watch as your four lines are pulled into a perfectly regular square. You get to kick back and watch as your four lines are pulled into a perfectly regular square.

You don't have to actually draw in much exacting precision. As long as you get close to what you want, constraints can be used to finish the job. That makes Sketchpad a remarkably powerful tool. But it doesn't mean the constraints are easy to deal with. The thesis version of Sketchpad already had a lot of constraints, so there weren't buttons dedicated to each possible rule. Instead, you had one button to apply a constraint. How would one specify the constraint to apply?

Well, slowly. The answer is slowly. Constraints were numbered. To choose a constraint, you had to enter that number into the so-called Toggle Register 25. This was a set of physical switches on TX2's control panel that, well, they were toggled to represent a numeric value, in binary, of course. These toggles were located near the CRT. Sutherland's thesis has this picture of him using Sketchpad, and you can see that the toggle registers are just off to his left-hand side. So you aren't getting up and walking to a control panel on the other side of the machine, but you are entering a binary number on a set of switches that's away from the CRT. That's not really fun. You may be asking yourself at this point, why did constraints have to be entered by toggle? Well, it comes down to

extendability. The sketchpad described in Sutherland's thesis is actually version 3 of the program. At least, it was his third iteration of the code. During refactor after refactor, it became clear that the code had to be cleaned up. So by version 3, it was possible to add new constraints quite easily. With this kind of extendable system, it just wouldn't be feasible to add a new button every time a new constraint was written. Hence, toggle register 25. I also want to point out something weird here.

Maybe you've already noticed. Sketchpad uses a pile of different types of inputs. We have the light pin for selecting, buttons for actions, toggles for parameters, and the zoom and pan, well, that's handled by two rotating dials. So, yeah, part of Sketchpad's interface were highly intuitive and very modern, but other parts were much more grounded in the 1960s status quo. A recognizable GUI was still years away. That's all the nice, visible stuff out of the way. Which means it's time to turn to the purely technical.

It should be plain to see that Sketchpad, just as I've described it, was a very complicated program. Once we hit the back end, things get even more complex. Not all of Sutherland's good ideas are visible to the end user, but they all affect how Sketchpad is used. The biggest aspect that I want to examine is data representation. That is, how did Sketchpad store and represent drawings? That may sound a little dry at first, I admit, but it is really the bread and butter of the system. This gives us a springboard to dive into Sketchpad's more powerful and, frankly, revolutionary features. The main data structure used in Sketchpad was the so-called ring.

At least, that's what Sutherland calls it. The most common name here would probably be a doubly-linked list, or at least, that's what I'd call this structure. This is a special type of linked list where each element links to both the next and previous element. The ring part just means that the final element links back to the first, thus creating a circular structure. The idea here, as Sutherland explains in his thesis, is to keep Sketchpad's data as flexible and, well, managed as possible. A looped and doubly

linked list has a few neat properties. There's really no end to it, which means it doesn't really matter where you pick the list up from. Starting at element 5 will get you on the same track as starting at element 0. The double links mean you can traverse forward and backwards. So if you start at element 5, then you can go to element 6 or element 4 just as easily. That's really nice when you're dealing with groups of related elements, like points that form a line. Additionally, it's really easy to remove elements from a ring. If you want to remove element 5, you don't need any additional information, you just need that element. All you have to do is

set element 4 and 6 to point at each other, and presto, you've deleted your element from the ring. What's nice here is you only need the information held within the element you're trying to get rid of. For a singly linked list, you need some way to find the previous element. But hey, as long as you're doubled up, that's not a problem. That's the main big data structure at play, but this leads to another line of inquiry. What type of elements is Sutherland storing in these rings? The short answer here is anything. The long answer is generic data, hens, and chickens. Now, dear listener, this is another part of the thesis that really threw me for a loop. One of the things I knew going into Sketchpad was that the software was an early example of object-oriented programming.

In my head, I was just thinking, sure, why not? You have little drawings on the screen. Those are like objects, right? Of course you'd call them objects. Nice! I assumed that the Sketchpad thesis would reflect this. It does, but it goes much deeper. The core of object orientation in Sketchpad comes down to these flexible data structures. What really threw me here was when Sutherland went from describing generic data structures to talking about chickens and hens. His thesis has a whole section covering data structures. It breaks down the ring structure, has tables showing how structures are stored in memory, and then gets into the elements. Each element is described as a simple in-register or

in-element structure. The register naming convention is a little weird on some of these early computers. In the context of TX2, a register is just an address element in memory. So this means that each ring element is a chunk of memory that's some length in. Each of these registers is used as a pointer. That's an address that references some other structure in memory. As long as two of those registers are pointers to other ring elements, then you're golden. You have your looped double list. The rest of the element can point to anything. I'm harping on this because it's important to understand. Sketchpad was designed to use flexible, generic data structures. Once again, they're just described

as a chunk of memory with some pointers. An apt comparison here would actually be how Lisp handles linked lists. A Lisp list has two elements, a data section and then a pointer to the next element in the list. There is a lot of flexibility there, but you're dealing with elements of fixed sizes. In Sketchpad, by contrast, you can cram in as much data as you want. You could have an element to describe a point which just needs an x and y coordinate, or you could have an element that describes an entire shape. You could even have an element that describes a constraint rule. This gets us to the barnyard. It takes a little decoding, but I think I can give a reasonable explanation. To do so, I'm going to avoid quoting Sutherland,

since his explanation is a little confusing. A hen is the zeroth element of a ring. the zeroth element of a ring. All other elements are called chickens. That might not make a whole lot of sense in the context of a circular structure. An alternative name that Sutherland uses is the key, which I think is a better choice. The idea is that a hen serves as a key for what a ring structure represents. The hin also allows the ring structure to actually have a first and last element, since once you hit the hin, you've reached the top of the ring. Alright, that's a lot of groundwork for a single quote, so here's the payoff. Listen to this and think about what we've been discussing so far. From the Sketchpad Thesis, quote, In the data storage structures, the separation of general and specific is accomplished by collecting all things of one type together as chickens which belong to a generic hen.

The generic hen contains all the information which makes this type of thing different from all other types of things. Thus, the data storage structure itself contains all the specific information, leaving only general programs for the rest of the system. End quote. Ring structures aren't just used for storing data. They're also used for organizing code and data that's internal to Sketchpad. We have all the information we need to actually rename something now. If a HIN is a generic descriptor that isn't called directly, that's supposed to be used as a template for actual code and data.

Well, you could also call that a class. That's one of the core features of object-oriented programming. That's the really deep part, but what about the shallow side of the pool? How does the OO approach affect users? about the shallow side of the pool. How does the OO approach affect users? Well, this gets to the whole importing objects thing that I mentioned earlier. Once you complete a drawing, you can import that into another drawing. But this isn't just a copy-paste. You're using one drawing as an object. Then you create a new instance of that object. It has all the properties of the initial object, of the hin, so to speak. But you can make changes.

You can move it around, resize it, rotate it, whatever you need to make it fit into your larger drawing. Here's the powerful part. Let's say you import a drawing of a bolt into your sketch. You might have a lot of bolts to place, maybe a dozen or more. After a lot of work, you realize you drew the wrong head. The bolt needs to be a hex and you drew it as round. That's not a big deal, though. You go back to the bolt drawing and make the correction. All the instances that are used

in your larger drawing are references. They're based off that overarching class. When you make that change, it carries over to all the bolts you imported. That's the most simple example, but we can already see how practical and powerful this type of design is. So, we have a graphical program that's built in a very object-oriented way. This is before graphics are common, and really before people are talking about object-oriented anything. But can we call Sketchpad the root of these technologies? What exactly did Sutherland's thesis lead to here? First off, I want to take a diversion into the language Sutherland uses in his thesis. There are three big words that I want to keep in mind when discussing object-oriented programming.

Class, object, and instance. A class is like a template. An object is really just a fancy table. Instances are a little trickier to describe, at least in this hand-wavy, one-liner kind of way. When you want to use an object or class, you instantiate it, meaning you make a reference to the object. There's some fancier stuff, but for now we can just think of it as a very nice reference. The bolts example shows instances of a particular object. In all regards, they're bolts, except the instances can have certain properties that are tweaked. In the example, those properties are all spatial,

location, rotation, size, that sort of thing. But in all other regards, they keep the properties of the Bolt class there instantiating. In the sketchpad thesis, Sutherland doesn't use this exact set of terms. He does use the word object when discussing drawings, and uses the term instance or occurrence when talking about, well, instances of drawings. So we have two of three accounted for, at least mostly. Sutherland does not use the word class when discussing these generic template definitions. He uses either the terms master when discussing drawings or generic when discussing the backend code. He also uses hin, but I think we can all agree that's meant as an illustrative name. Once again, I'm nitpicking here because I want to

highlight that Sketchpad had some modern ideas, but that doesn't mean it looks fully modern. Brand name classes don't show up until the middle of the 1960s, probably first in the Simula programming language. So just as with the GUI layer, we're still years away from a fully modern creation. That said, we're getting tantalizingly close. Sketchpad would serve as a major inspiration for the development of object-oriented programming. The thesis also earned Sutherland a PhD, in case you're keeping track at home. One of the most glowing endorsements of Sketchpad came from the aforementioned Alan Kay. glowing endorsements of Sketchpad came from the aforementioned Alan Kay. For context here,

Kay was one of the main developers of Smalltalk, which is one of the first big object-oriented languages. He also worked for Xerox PARC Lab during the development of the Alto. This puts him firmly at the forefront of object-oriented programming and the graphical user interface. During a 1987 lecture, Kay relayed this interaction with Sutherland. This should give you a good idea of how Kay felt about the whole sketchpad thing. When asked, how could you possibly have done the first interactive graphics program, the first non-procedural programming language, the first object-oriented software system, all in one year. Ivan replied, well, I didn't know it was hard. End quote. That's a lot. It also gives us a lot to discuss. In this one passage, Kay is proposing, implicitly, that Sketchpad was

the first interactive graphics program, non-procedural language, and OO program. The year part is really neither here nor there, just adoration. Is any of this true? And what exactly does this all mean? The interactive graphics part is a little weird to quantify. As discussed, there were already interactive programs using TX2's display. There was an Etch-a-Sketch ripoff that was written well before Sketchpad. Sure, it wasn't nearly as impressive, but it did exist. It was graphical and interactive. And, well, it was also a program. Sketchpad itself benefited from yet another early graphical program, Sage.

That system didn't really let you draw fancy pictures, but it contained all the technology to do so. In other words, Sketchpad isn't really the first interactive graphics program. That said, I get where Kay's coming from. Sketchpad is the first system that really does it all and does it right. That puts everything together in a coherent package. You get graphics and interactive display plus all of Sutherland's technical magic rolled into one program. That makes Sketchpad a much more influential program than TX2's Etch-a-Sketch demo. As far as the first object-oriented program, well, I think that's broadly accurate. Sketchpad is at least one of the first well-documented programs that acted like it was using object orientation. The name here is

an anachronism, since it wouldn't be coined for a few years, but we can apply it just fine. We have all the hallmarks, so I think we can go with it. Now, all good things come in threes, and I've been avoiding the middle honor of Kay's list. He claims Sketchpad was the first non-procedural programming language. What does that even mean? And is it true? If it is, why does that matter? The terms we use for describing programming languages are... I mean, just frankly, they're darn confusing. What sucks is that languages have many aspects to them. You can build up this giant overlapping diagram of languages that treat variables in certain

ways, or handle functions in other ways, have certain levels of coupling with hardware, or look certain ways. When Kay mentions non-procedural programming, he's talking about a paradigm. That's an overarching way a language is constructed. Think of them as strategies that languages use. Now, a single language can actually adhere to multiple paradigms, which makes this, well, darn confusing. So let's switch this up and drop the non part for now. We'll get back to that. A procedural programming language is just a language that uses procedures, aka functions, aka subroutines. There are small differences and implications between those names,

but in the abstract, we can treat these as synonyms. This paradigm encompasses everything from Fortran to C to Python. We're looking at languages whose most used grouping of code is a function, a procedure. These procedures are just tiny programs. They take inputs, they run a few lines of code, and maybe produce outputs. There's also some implications about scope. I also have to throw the little wrench in here that procedural languages function as state machines. With each line of code, you're changing some overall state. That's just a fancy way to say you're setting variables and messing with memory. These state changes will happen in sequence. In other words, one line of code comes after another. That paradigm, this state-based

viewpoint, is also called imperative programming. So there's another nice word to whip out and impress people. So then, what would a non-procedural programming language be? We're just looking for a language that doesn't use procedures and doesn't do the whole state machine boogie. Now, I'm pretty sure that when Alan Kay said Sketchpad was non-procedural, he just meant it was object-oriented. I have my qualms about calling any OO language non-procedural, though. In a purely object-oriented language like Smalltalk, you don't have procedures, at least you don't have brand-named procedures. Objects can contain what are called methods, which function in a way very similar to procedures, just, you know, more fancily.

Small talk specifically, and really, a lot of better-developed OO languages use message passing instead of procedure calls. That's where you tell an object that you want something, then that object services the request on its own. There is some nice set dressing there, but you're still essentially calling a procedure. Plus, despite how isolated objects tend to be, they still do the whole state machine thing. One line of code comes after another, so maybe K was getting at something else. Another option here is to look at a different part of the programming dream diagram. Declarative languages. I think this fits the bill much better.

A declarative language is usually defined in opposition to imperative languages. Instead of slowly changing the overall state of some set of values, you just declare what you want done, and the language does the rest. It figures out the details. Once again, as with almost every programming episode, this drops us into the Lisp v Fortran continuum. Lisp is, at least in part, a declarative language. You specify how a list is composed, or what operations you want carried out on said list to create some other list. Lisp then deals with the finer details. You're essentially writing logic proofs or definitions. Something like,

list A is composed of the union of lists B and C. That's how a declarative language tackles problems. In a more imperative language, you'd just take a different approach. You'd probably make an empty list for A, then execute a union operation, then set A to be the result of that operation. It's a matter of steps versus definitions, really. Sketchpad fits this concept very nicely. At least, if you choose to think of Sketchpad as a programming language. You define geometries and apply rules. You could do something very similar to our list example. Create a plate that contains flanges A and B, where each flange has its own definition and all sides are collinear.

That's very much non-procedural. Instead of defining steps used to create a drawing, Sketchpad lets you define what you want. Then the program does the work. But was Sutherland the first on this non-procedural bandwagon? Did he invent a new paradigm? Well, the answer to that is no. I don't think we can go that far. I mean, Lisp existed prior to Sketchpad. It even existed on the same campus as Sketchpad. That said, Lisp is technically a multi-paradigm language. You can write imperative code in Lisp. So while

there are declarative parts, it doesn't 100% buck the procedural realm. There's also APT to consider, but once again, that gets kind of muddy. An APT program consists of two parts, geometry definitions and toolpaths. The geometric part is definitely declarative. You declare that you want to make a shape and what that shape's composed of. In that way, it's similar to a textual form of sketchpad. I mean, it even served as inspiration for Sutherland, so maybe that's not too surprising. The toolpath, however, is definitely imperative. You modify the state of a machine. In this case, it's a very physical CNC machine. Defining the toolpath involves telling the machine how to move around, step by step. You still do the whole state machine thing, even if you don't technically

have functions. We could definitely go further afield and find some pre-Sketchpad oddities, but I kind of don't want to do that. That would get us to things like track, perhaps. It's fair to say that Sketchpad was one of the earliest declarative languages, with all of the caveats I've given, and I think it was probably the first purely declarative language, once again with all the caveats I've given. Now, you may ask yourself why I've gone off on this programming language tangent for a graphics program. Well, frankly, it's because I think it's kind of funny. Sutherland wasn't going into Sketchpad to develop a new language, not at all. He wasn't trying to create an object-oriented program. He was making a really slick tool for engineering drawing. He was using new technology

for a new application, and that happened to lead to some interesting new approaches to programming. Which brings us to the last big question of the episode. Did Sketchpad inspire Doug Engelbart to create NLS? We're right back to the top with this one. It's clear to see that Sketchpad had influence on Kay, but what about Engelbart? The story here actually is pretty complicated. The first touchstone for us is Engelbart's Augmenting Human Intellect. That's his manifesto on new user interfaces that was published in 1962. This makes the publication roughly contemporary to Sutherland's work on Sketchpad. Engelbart even mentions Sutherland by name.

Here's the full passage for context. Mr. Douglas Ross of the Electronic Systems Laboratory at MIT has, we learned by direct conversation, been thinking and working on real-time man-machine interaction problems for some years. We have recently learned that a graduate student at MIT, Glenn Randa, has developed the design of a remote display console under Ross for his graduate thesis project. We understand that another graduate student there, Ivan Sutherland, is currently using the display computer facility on the TX2 computer at Lincoln Lab to develop cooperative techniques for engineering design problems. End quote. So Engelbart knew about Sutherland's work. He even had communications with Ross, the creator of APT and also,

I don't remember if I've mentioned this before, but he was on Sutherland's thesis panel. So there was apparently some exchange of ideas, a pipeline so to speak, but this evidence doesn't point towards Sutherland inspiring Engelbart. It's more that they most likely knew about each other's work. I'm using augmenting human intellect as the benchmark here because this is where Engelbart lays out his initial research. This, with some added work and funding, leads to NLS. That's the big graphic system that is essentially the root of our modern notions of a GUI. In a lot of ways, NLS is an implementation of what's described in Augmenting Human Intellect. There are major changes, but the inspiration is already there. Was Sketchpad the inspiration? No, the timeline

just doesn't make sense for that. However, there is a deeper connection and a bit of a complication. Between 1962 and 1964, J.C.R. Licklider was the head of ARPA's Information Processing Techniques Office. He was one of the first backers of Engelbart's work, and much of the funding that helped create NLS came from the IPTO. When Licklider left, Sutherland took his place. Sutherland stuck around for two years. In that time, he was holding at least some of the purse strings that led to Engelbart's lab. Yet, once again, this is somewhat of an oblique connection. Sutherland and Engelbart knew of each other's work. Perhaps there was some influence, but if there was, neither of them ever discussed it in

writing, only briefly mentioning it. It's fair to say that Sketchpad was very influential, but I think the hypertext lineage that some articles try to outline just isn't there. Alright, that does it for this episode. Sketchpad was truly a marvel of programming, especially for the early 60s. Ivan Sutherland was able to use slick code and some really fancy hardware to make something totally new. His work would spawn many more developments, from object-oriented programming to improvements in computer-aided design. I highly suggest reading Sutherland's thesis for yourself. It gives a fascinating look into how object orientation can be implemented. That's something that's always confused me, since OO just seems so abstract.

But hey, what better way to start understanding the details than looking at object orientation's roots? That said, Sketchpad was still archaic in many regards. The full object lexicon didn't exist yet. The graphical interface was... strange. You don't usually have to turn physical dials to operate a computer these days. So while Sutherland did make great strides towards the modern GUI, it's important to recognize that Sketchpad was just one of many steps. And finally, there is the matter of the hypertext connection. This idea that Vannevar Bush inspired Ivan Sutherland, who in turn inspired Doug Engelbart, well, from my research, that's just

simply not true. The connections are there, but they're too tenuous. I know I shouldn't throw a totally new idea into the end of the conclusion, but I'm gonna. I think part of the confusion comes from the fact that object-oriented systems, relational databases, and hypertext, they all have overlapping features. This is something that came up a bit during my database series. These three systems can all construct something similar to a link. That one feature might evoke ideas of hypertext. Sketchpad in particular has another. Its graphical interface. One hallmark of early hypertext systems are their fancy user interfaces. So if you look at Sketchpad and see a nice interface and links, well, that does sound a lot like hypertext. And Sutherland's

work did influence some developments in that area. Ted Nelson has written at length about Sketchpad, so there is a real lineage. However, that line doesn't draw from Bush, and it doesn't extend into Engelbart's early work either. Thanks for listening to Advent of Computing. I'll be back in two weeks' time with another piece of computing's past. If you like the show, there are a few ways you can help support it. If you know someone else who'd be interested in the history of computing, then please take a minute to share the podcast with them. You can also rate and review on Apple Podcasts. If you want to be a super fan, then you can support the show through Advent of Computing merch or signing up as a patron on Patreon. Patrons get early access to episodes,

polls for the direction of the show, and bonus content. You can find links to everything on my website, adventofcomputing.com. If you have any comments or suggestions for a future episode, then go ahead and shoot me a tweet. I'm at Advent of Comp on Twitter. And as always, have a great rest of your day.