a16z Podcast - a16z Podcast: New Year, New Horizons -- Pluto!

Episode Date: January 1, 2017

What (on earth) does it take to get a signal to Pluto? Stanford senior scientist and astronomer Ivan Linscott, part of the team that ran the radio science experiment on the New Horizons probe, shares in conversation with a16z's Frank Chen all the nitty gritty details about their project using Ruse radio transmissions to gather info about Pluto. Listen in on exactly what it really takes to do so -- everything from commandeering old Cold War spy technology and plutonium to completing the entire mission on approximately 250 watts, and including other such highlights as a motorcycle riding, guitar playing, leather jacketed, tattooed FPGA fixer coming to fix everything when it seemed a lost cause, and the satellite going dark just moments before contact. From deep tech details to the drama of accomplishing such a difficult mission, this podcast is all about how, exactly, we sent a radio signal to Pluto.

Hi, everyone, welcome to the A6 and Z podcast. Today's episode hosted by Frank Chen is a conversation with Stanford senior scientists and astronomer Ivan Lindskot, and it is all about the juicy, nitty-gritty, mind-blowing details of how exactly you send a signal to Pluto. Ivan Linscott is part of the team that helped the radio science experiment or Rex experiment that went up with the New Horizons probe and will use radio transmissions to gather info about Pluto. He focuses on digital signal processing, radio occultation experiments, I don't even know if I Announce that right, observational radio astronomy and particle physics, and has worked at Dudley, NASA, and Uncetti. All of the work has in common listening for signals and doing radio science experiments in the outer planets.

A16, Z, research and investing team head Frank begins a conversation by sharing how he came across this work. So, I'm sitting at home, and it was July of 2015 when all of the amazing pictures were coming back from New Horizons, and I'm having a bunch of Wi-Fi problems. at home. And I'm thinking to myself, here we are. There's a probe billions and billions of miles away that's sending photos. And I have this huge Wi-Fi deadspot at home. Like, what is going on? And as it turned out, a couple of weeks later, I was talking with the fabulous Teresa Johnson. Shout out to Teresa, who is now a data scientist at Pinterest. But she studied with Professor Lynn Scott. And she's like, oh, I know the guy who helped design that system. Let me introduce you. And so I got introduced to Ivan and he started telling me stories about the design process and the mission. And it was

just so fascinating that I was convinced that you would be excited to hear Ivan's stories. And so here we are. I mean, maybe you can tell us a little bit about yourself and how you got involved in communication systems design. Thanks, Frank. I got to Stanford a long time ago because of my interest in the SETI project. We had been developing high performance spectroscopy. in the radio systems for looking at pulsars in small observatory, Dudley Observatory, in the east, and it's connected in New York, as a postdoc there. And one of the applications of this spectroscopy was looking for alien signals on the sky. And promoting that got me an invitation to be a NASA fellow, an RC fellow at the Ames Center,

which was then pioneering the work on the development of the study instrument. So that got me to Stanford working with Professor Alan Peterson, who was then leading the signal processing group in the Department of Electrical Engineering. And Alan and I became responsible for the 10 million point Fourier transform analyzer that we built for the SETI project and its prototype that has then since been used and think in other versions for the search that they've conducted ever since. But I haven't been involved since the late 80s when I started working with, seriously with the space sciences group and electrical engineering under Professor Tyler there.

And it was his interest to do radio science experiments in the outer planets, which attracted me very much. One of the things you can do with radio is like remote sensing. And their specialty, Lens specialty with Professor Herschelan at the time, was to do what they call radio occultation. It's a process by which radio signal was transmitted from one direction and, like, refractive. or diffracted in an atmosphere or off of rings or some other process on a planet and is received on the spacecraft or vice versa. And using that, you can extract the properties of the refractor. And in this case, we get from that we can recover the temperature and pressure profiles of the atmospheres of the objects. And so that was our plan for Pluto.

When Pluto mission was announced, initially there were several incarnations of it. And one of the early ones was in the mid-1990s. And Len, Tyler, at the time, said, you know, I have done every occultation in every planet in the solar system except Pluto. It was still a planet then. Right. Poor Pluto. And he said, this is going to require a novel architecture for the acquisition of the signal. It's going to have to be transmitted from the Earth, not the spacecraft.

It's going to have to receive on the spacecraft. It means the receiver is going to have to be modified. They don't like to do that. They like to fly things that have flown before. They don't like novelty. But if we want to do this, we're going to be able to. going to have to do the modification. And so we proposed in the early 90s to develop the technology. NASA has a program for technology development that is

applicable to future missions. And so they gave us the support to build prototype receivers that would do this, would be able to capture a narrow band signal and record it and send it back with the opportunity to extract those properties that allow us to do the inversions. That resulted in a sequence of proposals. And so we teamed with a group of Southwest Research, led by Alan Stern, to do this new Pluto mission, they would call New Horizons. We became the radio science experiment called Rex. That had the objective of doing the radio occultation of Pluto's atmosphere and Karen, which doesn't have an atmosphere. And to do what we call radiometry, to measure the temperature of Pluto using the radio

antenna, and to do an additional experiment where we bounce signals off the surface of Pluto, called a bi-static radar experiment that would characterize the surface to an even greater degree. And in the ensuing years, we had students developed the technology. We had them build it. We had them test it. It was integrated into the spacecraft. And you know, you pretty much know the rest of the story. We got our data back. We got an exquisite profile of Pluto's temperature and pressure. We did radiometry measurements and discovered some really quix. surprising things like the night side is a whole lot warmer than the dayside and we got the

bisatic signal which is itself or kind of a small miracle yeah that's amazing so funnest phd projects ever right to design the system a novel system that would fly for the very first time further than we've ever flown before is that true well the pioneers are out there further yeah this is the furthest radius occultation experiment that's ever been done and the furthest bistatic radar furtherist radar experiment that's been done so So in a way, there are a lot of firsts associated with this. Tell me how new horizons got its power. Well, because Pluto is roughly 37 right now, you know, 40-ish AU out from the sun, 40 times the distance.

You can't use solar power because the square of the distance. And so 40 squared is 1,600. So you get one 1,600th of the power of the sun that you would get at the Earth's orbit, and that's not enough. So common these other, of these deep space missions is to use a electric source called an RTG, a radiothermal electric generator, which uses radioactive nucleides to generate the heat, which is presented to a thermal pile that generates the electricity. We use a radioactive element called plutonium to generate the heat, and that generates about 250 watts for us of power. Yeah, so if you think about 250 watts, that is a very modest power budget. So, entry-level PC without a graphics card, probably 200-watt power supply? Probably, you know, it's a couple of light bulbs, you know, yeah, and it's not a lot.

Yeah. But you run the whole mission on that. I mean, everything. It's amazing. Propulsion and telemetry and communication, all the scientific experiments off of 250 watts. It will turn out. You can't run them all at the same time. Oh, right.

One at a time. Am I steering or am I doing science? Well, you can do four out of five, I think of the payload instruments. And so one of them has to be turned off. But fortunately, or maybe not for the instruments, they're body-mounted, unlike a lot of other spacecraft. This means that all the instruments are not all facing the same direction at the same time. So you can only use one or two at the same time, as it turns out. Right.

And be pointed in the right direction. Yes, yes. Now, plutonium has this property that it degrades over time. It's radioactivity. And so did you need to figure that into the design of the system? Oh, sure. There's a bunch of plutonium isotopes. The longest-lived ones are about a quarter of a million years,

but they're also the ones that don't generate as much heat because they're not popping off as much, okay. The ones that are the hottest are the ones that are the shortest-lived, and they're about 84 years. So what you want to do is you want to get a fresh supply of plutonium, sock it on a spacecraft, send it off so that in the first year or so you don't lose more than a few percent, a few 10 percent of the power.

In our case, that process got started. there is a plutonium processing facility that the U.S. Department of Energy maintains, and that there was, in about halfway through the process, there was a leak that was discovered in the radioactive leak, and they shut down the process of production indefinitely. That doesn't sound good, right? Because you have a window that you can get to Pluto, right? We had a launch window that was narrow. For a lot of reasons, one was we were looking for a gravity assist at Jupiter, which did boost the, or arrival time by about four or five years. It also was true that we were afraid, Pluto has sort of

an elliptic orbit. At the high point in the orbit, the apogee, it's colder significantly than in the perigy, which is the closest part, and it had just passed the warmest part. And so what happens if the model for Pluto at the time was kind of like a comet. So when it got warm, the atmosphere would bloom. When it got cold, the atmosphere would collapse. And if there was going to be an atmosphere there that we were going to measure its temperature and pressure, we'd doggone well better get up there before too much time it passed because it was very likely to have to collapse sometime soon. So there was a lot of pressure on the mission to design, build, and launch. Yeah. So you have to build this thing that's never been built before. You're racing against time

because otherwise the atmosphere you want to observe isn't there. And then the radiation processing plant springs a leak. That's right. And then they tell you this might not be possible. So we had at the time the project had assigned a liaison to the Department of Energy within the program who was being, the spacecraft was being developed by APL, Johns Hopkins Applied Physics Lab. And Glenn Fountain was the person who was in charge of this process, of acquisition of the RTG, and mitigating problems that would come along with getting the license. You need a license for this from the Department of Energy. And it has to go through a whole lot of layers, including a congressional.

and approval and environmental impact studies and people who had protested the launching of plutonium before because other spacecraft had had it but there were a lot of protests about putting that on a rocket and sending it up in the air. Yeah, what could possibly go wrong? Yeah, what could go wrong with that? So we had to prove that if something went wrong on launch

and the payload exploded, which they do... Every now and then rockets explode. Well, and they actually put that capability in there in case the thing is going in the wrong direction, after a launch failure. So you want to be able to mitigate that threat, you know, boom. So you have to prove that if you do that, that the plutonium won't disperse, it'll fall down and intact, and in such a way that it won't be potentially harmful,

all that. In addition, now the thing has been shut down. And so Glenn, Glenn, to his extraordinary credit, got into motion, and he started discussing with the U.S. State Department, alternative sources of this rare and precious quantity called plutonium, which you probably understand is actually one of the reasons why it's scarce is that it's nuclear fuel. In addition, it's nuclear bomb fuel.

So the control... Not that many people on the planet can make it, thank God. And there are quite a few that do, that we don't want to, and some that have done so that we had an excess of it, the Russians, under this former Soviet Union, made a significant amount of plutonium, and they had it in storage and it was my understanding

that what Glenn discovered was that the Soviets or the Russians now were anxious to maybe use it as a marketable option that they were looking for cash they desperately needed the cash they would have possibly entertained an buy offer from a variety

of vendors and the U.S. did not want that to happen because they didn't want it to fall in the wrong hand. So Glenn, brokering through the U.S. State Department, got the U.S. to make an offer to the Russians to buy their plutonium for this mission. Unbelievable. For the missing piece of it.

It wasn't the entire piece, but it was about half. So that was brought over to the processing facility that could package it, which was not shut down. And they did that. They packaged it, put it up, tested it. Flynn got the licenses. It was all under extraordinarily short views. The last time that that happened for the Cassini mission, the license itself took like three years, maybe pushing four. and we had to have it in under six months.

That's amazing. And it was amazing. I just didn't think our odds were very good for that to happen. Nobody really did. And tell us more about Glenn. So what was his role on this mission? Well, Glenn became our program manager. The program manager of the mission at the time became ill,

and they were looking for somebody to step into John's ample shoes. And Glenn was asked, and he agreed. Glenn brought to the mission that same quality of statesmanship and problem solving that he demonstrated in the brokering of the plutonium, and it really eased so many problems that occurred along the way that had to do with funding irregularities, recalcitrants on the part of the U.S. Congress and the White House and all of those other problems that pop up day-to-day that you would otherwise, any one of which could have stalled this for too long. Glenn made work of, and I think we all credit him with making this possible.

That's fantastic. And so you get part of the plutonium that you need, and who was the supplier of the last part? Well, that was the Department of Energy. Okay, good. So now we've got enough plutonium. Actually, we were still a bit short. And the DOE said, well, you know, we have put aside a few extra pellets for the next space mission. We might be willing to trade those for some new ones that would come along.

And the problem is that the ones we have put aside are now a bit old. Right. They don't have so much juice anymore. You won't have the power that you expected. We had maybe thought we'd start out with like 300 watts, and we didn't. We started maybe with 280, 275, and by the time we get there, it's pushed down to 245. It was a succession of compromises and brokered deals, but eventually it all kind of worked. Yeah.

Well, let's work our way back to the impact on the communication systems, right? Because you had assumed a power budget, assuming fresh plutonium from the Department of Energy, and you got kind of tired plutonium from a variety of sources. and now you have a much lower power budget. Right. We were initially offered a chance to implement our new technology for acquisition of the signal that was transmitted from the Earth to the spacecraft. Like I said, we had a sponsored research development program. We knew how to do it.

We thought we could fit it into a signal processing system, but it would require in the existing technology of the early 2000s. These have to be radiation-hardened electronics because, after all, you've run a spacecraft and not two meters away is this riotous source of radioactivity pumping out gamma rays and neutrons like crazy. Not good for chips. Yeah, you have to have a radiation-tolerant implementation. And that was possible in a signal processing element

called a FPGA, which are field programmable gate arrays that Actel was making. And they were based on a technique of hardened by design, but also what they call triple redundancy. You have each element in each gate in the array is triply implemented, and you vote them in pairs. So if all three agree, fine. If two agree, you chose that one.

Right. And if none of them agree, well, you've got a problem. The minority report. That's right. That's right. But it will turn out that it will be the case that you'll get three or two, essentially all the time. So those are limited in capacity.

At the time, I think you could get a million gates in one of the highest density act tell you could get for this case. which has to be spaceflight qualified. So we had been offered the opportunity to implement in two of these. And as you say, Frank, as the power budget was reduced and the scope was re-evaluated, they came back to us and said, well, guess what, we really have to have one. It's just one. Wow.

All the way to Pluto. You've got one. You've got one. FPGA. Now, we knew that we were, at the time, about 50-50. Now that we could put half of the design in one and half of the other, and they could talk to each other, would be all just fine. And that was good from a lot of reasons because one of the design guidelines from the mission is that you have to have a margin of gates in these FPGA that is high enough to give you assurance that the implementation in the gate is going to be successful.

One of the biggest problems in implementing an FPGA design is routing. And being able to successfully route to all the gates in this million point of, gets exponentially more complicated, as you get closer and closer to the top of the line. And so they say, got NASA guidelines, as you need at least, we would like you to have 25% margin. Minimum number is 15%, which means 50% of a million gates, 150,000 gates are just sitting there unused.

And we were fine. We had 50%. Okay, now we have to stuff it into one. And we said, well, we're not sure we can do this. And they said, well, I don't think you have a choice. So I had a student at the time who had designed the signal process for this application.

This is Kamakshi, Shiva-Ramakshin, Kamakshi's wanted to take on a challenge. We said, well, this is the challenge. We've got a signal that's coming in in this big wide bandwidth. It's 4.5 megahertz. And this actual signal is about kilohertz, which is basically 10,000 to one reduction in bandwidth. And we've got to make that reduction. We have to throw away all that other stuff

and pick that one band that we need. And we have to do that in such a way is to, guarantee linearity in the process. That filter has to be perfect. It has to suppress the output and noise and inside the band has to be flat and has to be linear.

These are called fIRs, finite impulse response filters. But a 10,000 element, FIR, is unimplementable. There is no algorithmic stability that you could possibly find in any computing technology, even today, that would do that.

So Kamakshi's task was to figure it out and do it in a way that was not computationally, expensive. And she did. She discovered a old radar idea by a RCA employee, Glenn Hogamire, who was working in the 1950s and 60s when they didn't have a lot of transistors in a chip and they had vacuum tubes and they were doing radar on planes. And they wanted to be able to reject the clutters and the jamming. And so they wanted to be able to filter. And so Hoganire had developed these computationally efficient techniques for filtering linearly. And we using vacuum tubes and Kamakshi's design was, I think this could work if we did it this way. So she did. Then I had another couple of students that tried to take that and drop it down into a programmable device, FPGA.

And well, So Actel gives you all of the tools. And they'll even give you a course in how to do it. And they give you the development environment and they give you program languages and they give you simulators. They give you the routers. They give you the emulator to test it. A couple of students, one after the other, went through that whole process, got it down into the chip. I actually bought some chips and the programming the stations and we burned them in and we turned them on and we ran a signal in and it didn't work. Oh, no.

We spent the better part of a year not getting it to work. Wow, that long. Yeah, yeah. And after, and not it wasn't a whole design. It was just a small test piece that used maybe a tenth of the hundred thousand gates. It didn't work. There was something fundamentally broken in the process of how that, language assembled into code that was then dropped onto the device.

It's something fundamental. We didn't understand. After a year, the program manager that oversaw the entire spacecraft, Glenn was the payload guy, and there was a manager at Southwest San Antonio, which had responsibility for the whole program. He's getting worried. He's saying, Stanford doesn't look like it's going to succeed. We're going to have to do something. So he sends a SWAT team in to review what we're doing.

And so already my hackles are up, right? I'm real prickly about this, right? The guy who was head of that evaluation was a former Stanford graduate student but not too long ago was now the payload chief engineer and Mark Taffley, and we became close friends quickly.

It was amazing how that compatibility seemed to kick in. So Mark went back and he recommended to Bill Gibson, the program manager, you know, that Stanford needed help. But they didn't need, it wasn't probably a good idea to take the whole project away and give it to somebody else. So how can we help them? Well, Southwest had a FPGA programmer for spaceflight's qualified solutions. Mark Johnson.

Mark's a motorcycle riding, guitar playing, leather-jacketed, tattooed. I have this great mental image. He's out of Texas, right? So Mark shows up and we, you know, wants to know what's going on. What are we doing? So Kamakshi's there, you know, we give them a story, we tell them what's going on. Two things really important happened. One is Mark became really fond of Kamakshi.

Always helps. The other was that Mark says, you know, I have a very strict discipline about how I code. There are certain choices you can make in the language of the code that are absolutely guaranteed to fail. He said, you only know that woodsy lore by, you know, a lifetime of, trial and error and mostly error. But these are just, this is a discipline you have to have. He said, students, PhD students in Stanford in electrical engineering are too clever. They see the opportunity to recast an algorithm into a more efficient form and they'll do that. They can't resist doing that. Stee said, that's the problem. Mark took over the code and with Kamakshi's interaction, they

succeeded in getting the first test module to work and then the succeeding module is one after The other, and when he was done, he had, I think it was 25 gates left over in the, out of the million that was in the FPG. 25 gates, not 25%, 25%, less than 1%. So we went to the project and said, okay, here's the deal. It works. It works. It works. It fits.

We've demonstrated it across the board. There are no test procedures that show any vulnerability. We've stressed it, temperature, radiation. done all that stuff so you can you got a choice you know you can take it as it is and can wave the margin or give us the other chip so they waived the margin then they said well actually there are a couple of other functions that this ought to have do you think you can fit them in and so barry says I'll take a look and I said are you kidding he said no I think so I think so when you so he did and they were a couple of just simple protocol acknowledgments on the communication link that they

wanted to see to make sure that the thing was handshaking correctly. So he was done. He had five gates left. Five gates left. And that's the way we flew. So the FPGA on the mission to Pluto has five gates. Five under, you, on non-use gates. The Lord knows how that actually the routing worked. But again, that was Mark's magic, be able to kind of groom the routing. You do these, you do these nets first, and then you do the next net set. And you just sort of layer it in. You don't give it all at the same time. And you just try to slowly ease its way into completion. It's hurting cats, but you'll get them all into the corral. Let's back up a little and talk a little bit about how the probe actually communicates its signal.

So where do the signals go? Where are they received? So we transmit from the Earth with the Deep Space Nets stations, which are three of them that ring the Earth, and we use two of them because at the time of the encounter, Pluto is up above the horizon for Goldstone, California, and Canberra in Australia. They have antennas that are two sizes. One's a 70-meter diameter, and the others are 34s.

They all have these 20,000-watt transmitters. So during the occultation portion of the fly-through of the encounter, we had four of those transmissions going simultaneously. Actually, five, and I'll tell you about the fifth is the bi-static one. So the 70 and a 34 from each of the two stations, transmitting at frequencies that are within this one kilohertz of each other, spaced about 100 hertz apart, to the spacecraft. What's going to happen is as the spacecraft flies to Pluto and then pass,

and it's at an angle and it flies into the shadow of Pluto as seen from the earth, and so that's the occultation. On the way in, to the shadow, and then coming out on the other side. And in those moments before ingress, as we call it in egress, the refraction of those four signals through Pluto's atmosphere has the effect of shifting the frequency of the received signal. And that's because the direction of the ray bends slightly and you're getting the Doppler from a slightly different direction

than you were before. So what we do is we capture that central 1,000 hertz and we sample it into the waveform that is present in the signal. And the waveform has got a lot of amplitude. In other words, we've sent 20,000 watts on an aperture that's about 70 meters in diameter. So the instantaneous power in the beam is huge. It's over a million watts per square meter.

At Pluto, which is 3.7 billion miles out, in a system that captures this, those waveforms are sampled with noise that's 1,000th of the amplitude of that signal. So it's like you see a sine wave, dot, dot, dot, dot, dot, dot, and you're sampling it many times per cycle. Each dot is a sample value, has noise associated with it,

associated with the receiver's noise and the sky noise and the noise from coming down the radio path from the earth. You're going through three and a half billion miles of solar wind and plasma irregularities. It's extraordinary that all of that happens and you still get a signal where the amplitude is noisy, only a thousandth of the waveform.

Now the power, which is square of that, you square the amplitude of the voltage squared as power. So a thousand squared is a million. So we got signals that were a million to one. one above the noise power. It's at Pluto. Yeah. At Pluto.

So what you can do with that is a lot. The way we sample it is we have a clock that's running the sampler on the broadband channel. It's running a 10 million samples a second. That clock is derived from an oscillator on board. That oscillator itself is what we call a USO, an ultra-stable oscillator. That oscillator is quartz. It's a slice of quartz, encapsulated in a glass sarcophagus, you know, carefully suspended and thermally. and thermally insulated and kept at temperature barely changing by a thousandth of a degree.

Wow. And as a consequence, it doesn't change its frequency by one part in 10 to the 13th. That's one tenth of a trillionth of the frequency. That is an accurate oscillation. So what I have is I have a million to one in the steel noise ratio sample to a precision of one part in 10 to the 13th. Basically, you can measure a gnats eyebrow at a billion miles with that thing. And that's what we did. The oscillator itself is its own story because U.S.Os were commonly implemented by the Department of Defense during the Cold War.

They were used for surveillance. If you ever read Blind Man's Bluff, it's about the submarine espionage that was conducted during the Cold War. And using atomic submarines, the U.S. would, and I'm not telling any secrets. This is in the book. They found the undersea cable that the Russians had laid from Vladivostok across the U.S. the bay, and they snuck in under the Soviet's surveillance, and they put a box onto the cable that had been designed by MIT, Signal Processing Lab, which would sample the fluctuations in the voltages that were induced by the communication traffic in the cable.

That's awesome. And then they would come back months later and retrieve the box and put another one down, and bring the box back to the de-scramblers. They had a de-scramblers. There's a bundle of cables with, you know, dozens of lines of communication, sometimes running, you know, simultaneously. They had to de-scramble it. They had to decode it. They had to translate it. And they basically knew what the Russians were talking about, you know, for years. Unbelievable. By putting a box over the cable. Over the cable. They didn't, they didn't puncture it, right? They didn't, there wasn't an invasive monitoring because it would have caused the cable to leak and fail. So you couldn't do that. Right. It was, it was, yeah, I count that, you know, among.

the top 10, I should be known, stories of all time. Yeah. Anyway, to do that, you needed the ultra-stable oscillator. And I maybe had rambling on here, but because the Cold War, when it went over, when that happened, the former Soviet Union was disassembled, the market in U.S.Os disappeared. We didn't need them anymore. We didn't need them anymore. There was really very few people that even cared about having a part in 10 to the 30.

They were part in 10 to the 14th then. Right. Anyway. So here you are in the year 2000. And we want one. One of these U.S.Os, where do you get one? Right. And it was the same story like with the plutonium.

We started looking for suppliers they didn't exist. There was a French company that was making them, and they had gone bankrupt. There was a U.S. company that was making them, they'd gone bankrupt. Interestingly, at the time, APL again, had a frequency laboratory, high-frequency stability laboratory, that they had been kind of encouraging this technology internally. And they had a program then of exporting technology to startups, and they thought, well, this might be a good opportunity to do that. So they did.

They transitioned the capability that they had to a small startup, and they started making these things. Wow. Gave them like three years. And so here we are, six months before launch. We have to actually have one of these things to integrate into the payload. And that company delivered five. Wow.

We tested the five, two were okay. Wow. Two met specs. And they were put on the mission. Unbelievable. So we've got one of these like incredibly rare oscillators. Yeah. We've got the sort of mix of old and new plutonium.

We've got the FPGA that's got five gates left over on it. And you launch the sucker. Right. Like so now we're going. And then so it's going to take years. It's almost a decade, right, before it gets to its destination. And so you're testing the communication all along. Yes.

Every, this is called the annual checkout. The spacecraft's in hibernation. Once here we like, wake it up. We put, we exercise a set of tests. We transmit to it. We do performance evaluation like we were going to do the occultation's tests, and we make sure that everything's running okay. And so once a year, for the nine years, we went through the annual checkout, plus we did two rehearsals.

We ran the whole process of occultation and radiometric measurement in rehearsal. Yeah. And then somewhere along the way, am I right, that you lost communication with the satellite? At the very moment, almost of the encounter. Wow. So you're almost there. We're almost there. And it checked out year after year during your yearly checkouts. Everything's hard.

And then you're almost home. There's a message from the mission operations manager. Who call her mom for mission operations manager, Alice Bowman. Yeah. And so mom calls. That's not a good sign. So Saturday morning, we're about to go to Baltimore, my wife, Margo, and I, for the encounter. A lot of the members of the team are already there.

And it's the Saturday morning we're having breakfast. And I don't normally even open my laptop. to see messages, and I do. And there's one from Alice, and she says, we have a conference call in an hour to discuss the spacecraft went safe. When the spacecraft goes safe, those are not good words, because it meant that something went wrong, and the spacecraft tucked its head under its wings

and shut everything off and tried to find out where Earth was and turned on a tone that said one of several things happened, and you need something to figure out what it is. is. Wow. And so you joined this conference call

and the satellite is nowhere to be seen. And they're trying to find it. Well, first

of all, they were fortunate to get, there was relatively easy to acquire the

beacon. And the beacon said that there had been a fault in the onboard

computer. And without knowing exactly what that was, the team needed then to establish

a set of protocols as to how to go and wake up enough in the spacecraft. Give us

the commands to wake up enough functionality to look at the error log and transmit the air log back so that they could figure out what happened. And so they said they'd done that. They would, they would do that. We'd get back to us and they in a few hours because it's a four and a half hour light travel time. So essentially round trip is nine hours. So basically in, we'll give us a couple hours to figure out what's going on. We'll have another conference call in 10 hours. And in 10 hours it turned out that

the computer had faulted because it encountered an overload, a timeout, actually, that occurred from the fact that a process had starved out the watchdog timer and the clock. So something had taken over that had taken too much time, and they weren't sure what it was then. It wouldn't take too long to figure out what it was. It was the next iteration of the communication. So in the next 10 hours later, what we knew was that the computer had been taking an image that was, taken earlier of Pluto against a small image against the dark sky. It was not very big. And they were compressing it to be using a JPEG algorithm. And it was going to be then compression and sent

and telemetry. It was the compression algorithm that wasn't done when they had sent a command to update the onboard flash memory with the up-to-date instructions as to when to do what. So it turns out that the computer wasn't finished. The command to update Flash was received. It was given a very high priority, the highest priority. It said, I don't know what to do, and it went safe. It raised the fault flag, shut itself down, went safe. So we're all thinking, why did that happen?

Who could possibly have screwed up and made that happen? That isn't, that's not something that's supposed to have happened right now. That shouldn't have happened right now. We know what this stuff does. Why is this a problem? And why didn't we find it when we ran the rehearsal? In other words, six months ago, we did exactly the same thing. We ran exactly the same programs.

We sent the exact same update, Command to Flash. Why wasn't it a problem there? It turned out that six months ago, the picture of the sky was dark. There was nothing but black in a few stars. So the JPEG compression took a whole lot less time. It was done when the Flash commands came in. This time, it wasn't. Unbelievable.

All the stars confused. the algorithm just enough or delayed the algorithm just enough to get into a race condition. Yeah, nobody had seen that, which of course in the course of spacecraft missions, you come to understand that things always go wrong. It's not for the faint of heart. That is amazing. And so you managed to reboot the computer? Computer was, yeah, restarted. The real concern was the we're on a timeline. It was everything on the spacecraft's run autonomously. And in order for the encounter sequence to proceed with the science that we intended to obtain, that timeline needed to start only, it was like 30 hours from when the fault occurred. So to their

extraordinary credit, the operations team was able to do that. They restarted the timeline just moments, minutes before it was needed to get it on track to do the encounter in the way that captured all of the scientific opportunity. So we were minutes away from not getting any of the pictures from screwing up completely yeah unbelievable well not completely because what it would have done is we'd have sacrificed some of the early scientific measurements for for the later ones all right for those of you in the listening audience go now google new horizons pictures and much deeper appreciation hopefully when you see those pictures realizing how close we were to not getting any of those so one other thing i'd love to talk to you about those are an amazing set of stories so

thank you for sharing them i'll talk to you a little bit about the dish at stanford the big dish The big dish, right? And so most people think of it as sort of a landmark on your hike. Hey, meet you with a dish. And it is. And it is. They'd have torn it down if it wasn't a landmark by now. Yeah, exactly.

Exactly. But it's actually being used in some different experiments. So tell us a little bit about that. Right. From time to time, the dish is actually in the stewardship of SRI, SRI International. They were partnered with the team at Stanford Electrical Engineering in the 1960s when it was proposed as a facility to support initially. an investigative technique that the U.S. Navy and Department of Defense was interested in for communication over the horizon.

They were interested in talking to ships at sea that were over the visible horizon, hoping to get out significantly maybe to as far as Asia. And in both directions, I mean, you want to be able to send a signal over and you want to be able to maybe listen over the horizon as well. That would have, you know, listening at that time, of course, a very popular, yes. So there were ideas from, again, Alan Peterson and his colleagues at the electrical engineering department that suggested that there were techniques for taking advantage of propagation of phenomena that would allow you to, for radio signals to refract over the, diffract over the horizon and be able to get something. So they figured they needed a big antenna and they needed transmitters

and they needed to have it high enough to be able to point, and be close enough to Stanford to actually use it, from a campus perspective. And so they proposed to the U.S. Navy to build this, and they were partnered with SRI to do so, and they got the funds to do that. It would then emerge that there was a subsequent interest from NASA to investigate the properties of the solar wind

that measure the total electron content in the solar wind out through the solar system. And that meant that because the pioneer spacecraft at the time were being developed and sent into the inner and hour solar system. If they carried properly, you could do this if you have two frequencies. And so you build a spacecraft with receivers that can operate at two frequencies, ideally an octave more apart. And so they did that, and they built facilities into the dish to transmit and receive on those frequencies.

And so for years after the experiments to do over-the-horizon work faded away, there was a long epic NASA work that was done. with really characterizing the solar wind throughout the solar system. It was an epic body of work, and they did a wonderful job with it. That's fantastic. And then after that period, did they shut it down? Yeah, by the late 1970s and the early 1980s, that work was largely complete. That task actually had been taken up to some degree by the next generation of spacecraft

that had their onboard plasma instruments, and they were able to do sample in situate the conditions of the solar wind. And so it became an unnecessary task, and they stopped that. There was very little else that stepped in, largely because there was a transition in personnel and capability. The interesting aspect, I think, of operations at the dish was in order to facilitate the tracking. In some cases, they wanted to track satellites that were Earth orbit, and so you want to be able to follow for low Earth orbit. you want to, as the thing rises above the horizon, you want to be able to, it's an altitude azimuth drive, which means that you have the ability to go up and down and rotate in azimuth

and then up and down an elevation. So you point it in azimuth at the horizon, and the thing comes up and you start to raise in elevation and move slightly in azimuth, and then as it gets close to the zenith, you're suddenly rotating very rapidly in azimuth and then coming down again. So the azimuth rates have to get high enough,

and you have to take this giant structure, and you have to twist it fast. In the 1960s, there were no electric motors that could do that. Electric motors are great if they're running fast. And the transmissions that you would need to gear them down so that the dish which is rambling along were impractical. Even to use those would mean that you'd sacrifice pointing accuracy. It's a big dish, the pointing, the beam is tiny.

You have to be able to control it to a fraction of a degree. You couldn't have done it. You can't do it with electric motors. But they found out that you could with something else. Otherwise, they wouldn't have built it. There's something else is hydraulic motors. Our hydraulic motors are very good at having very high torque at low angular velocities. And the hydraulic motors that were best designed for that were the anti-aircraft gun controls on U.S. battleships.

Right. And after World War II, those were being surplus. Oh. And so what they did was they got two of them. Well, maybe they got a few more for spares. and they stuck them into the altitude azimuth drive system for the dish. The hydraulic motors are a very peculiar beast. They use very high pressure on a set of pins that push against a off-center plate.

And what you do by differential pressure is you push that plate and it rotates. It's a funny concept and it's hard to explain without images. But the key ingredient is high pressure. And one of the problems with this battleship is that, you know, You know, you're under fire, and one of those high-pressure hydraulic lines is cut. The line becomes a weapon. It will spray tens of thousands of pounds per square inch oil around the control room. That doesn't sound good.

Yeah. It was a very dangerous job to have, which was the anti-aircraft control guy who sat in that seat, moving that thing around. Because if something went wrong, you probably weren't going to survive it. Anyway, how they got approval to stick that on the dish. On the dish. In this era of litigious conservatism, I don't think that would have ever happened. But desperation being what it is and innovation being the better part of valor, they did. But by the 1970s, the technicians that knew how to prepare that were retiring. The need for repair

was increasing. And so the general consensus was, we really shouldn't be running this anymore. Right. And so the combination of lack of objective with the age and the growing sense of risk meant that there really wasn't very much you should be doing with that dish. Yeah. So they shut it down. I remember when I was on campus in the 80s and 90s, it wasn't being used. It was just a landmark. Right. But then you had the idea that maybe we should be using it. Yeah, along about the same time I stopped work with SETI, I still carried that nagging sense that, gee, I really should be looking for some. something anyway. And so a dish is sitting up there, you know, fallow. And I started asking around and I was led to SRI and a colleague over there, Mike, last name is Cousins, Mike Cousins,

who was their, like their chief engineer that knew most about it and had been using the dish up until, and still did from time to time. So Mike and I developed a really lifelong friendship and the desire to keep the dish alive. And so what we found was that there were a couple of things that were in need. Clearly, one was the control, and the other was the receivers. The era when the dish was used, those receivers could be relatively low performance.

They were good at the time, but they were unsuitable for listening for like alien signals on the sky or looking for subtleties within the galactic background. And so we needed to demonstrate that if you had the right receiver, you had in this environment enough sensitivity to, with that dish to make a difference. One of the big concerns was that that dish is sitting fully visible from the Bay Area, one of the highest sources of radio interference that you can imagine.

And so we were necessarily concerned that even if you had the most sensitive receivers that they would be swamped by the interference in the environment. Indeed, that was true. And so what that spawned was a succession of research into the means of isolation and protection, really excision. you eliminate without destroying what you're looking for, the unwanted interference? And that became a research project for me, a couple of students that I had pioneered some very nice solutions, as well as Mike's ability to develop, to implement those. And it was Mike, though, that heard about electric stepper motors. So this would replace the hydraulic motors.

To replace the hydrolex. So what happened, another kind of wrinkle in the technology story, was that over the era from the 1970s to the 1990s and 2000s, the progression of the disc drive industry to ever-increasing capacity had led to the sophistication of these electric stepper motors for controlling the drive on the oak of the magnetic readers going in and out. And they had a step with increasing precision, and they had to do it through precise rotation of the motor.

And it had to do it quickly, the quicker the better, because they were increasing speeds and down. densities. And so what happened was that this drive industry spawned a technology for the electric stepper that was for purposes of industrial control. And so now instead of something that was a cubic centimeter in size in your little disk drive, you could buy something that was the size of a Volkswagen and would turn the dish. And they did. We replaced the hydrolex with electric steppers and the dish woke up. We were able to do not only a high accuracy, high speed tracking on the sky, which we were involved in a couple of failed spacecraft.

There were a couple of spacecraft were launched, and then they stopped talking to the ground, a couple by U.S., a couple of by NASA, and a couple of our colleagues in Britain. The realization was, if the thing is alive, but the radio system is broken, so the transmitters say died, but the receiver is still alive, maybe we can still figure out what happened. Maybe it's just stuck. We could send a command to unsick it, but we need to know. what's going on. So if the thing is still alive up there, then the telecom system is partly awake and the reference frequencies from their oscillators are leaking out from the spacecraft. The thing

is that the leakage is tiny. It's a millionth of a watt coming out of the spacecraft in orbit a thousand kilometers away. Yeah. So to find that needle, right? It didn't take a lot to realize that the dish has a sufficient aperture and sensitivity to see a millionth of a watt at a thousand kilometers. That's amazing. Strongly. I mean, things stuck up like a sore thumb. Norad was tracking the space junk.

It said, here's the ephemorist of the thing. And we programmed that into the tracking computer. We tracked it and turned on the Doppler compensation, and bam, there it was. And there it was. You found these satellites. It was still alive, yeah. Unfortunately, I think except for one, you couldn't fix it.

Right. At least you knew it was there. At least it was there. Yeah, that's fantastic. Well, Professor Lin Scott, thanks so much for coming to share these stories. They're amazing stories, and I'm glad we got to share them with a broader community today. And I appreciate the invitation so very much. It's not often, you know, we get to ramble about this stuff, and it's a pleasure to do that.