Embedded - 57: Engineering on the Run

Episode Date: June 25, 2014

Ken Milnes talked to Elecia and Chris about his career developing augmented reality for sports broadcasting. SportVision MLB Stats

Welcome to Embedded, the show for people who love gadgets. I'm Alicia White. My guest this week is Ken Milnes. We're going to talk about hockey, football, car racing, sailing, producing TV shows, and meeting Nobel Prize winners. Hi, Ken. Thank you for joining me. Hi, Alicia. And Chris is co-hosting again this week. Good afternoon. I listed quite the list of topics.

I knew that talking about sports is safe because you worked at Sport Vision when you were working on NASCAR racing. Yeah, NASCAR, that's where we met, Alicia. We were developing a GPS system for the NASCAR race cars And we hired you and your company to develop an IMU system for the navigation Well, it wasn't really my company, those were my crossbow days Yes, it was And this was made fun of in the movie Cars Where you put the little arrow that shows which car is being talked about.

And then in the movie, they got it all like the arrow was wrong. Oh, that was the work of- You must have hated that. I did. That was the work of our director on the program, Artie Kempner. And well, our system wasn't perfect. We had little errors where the pointers might point at the wrong thing. And I already thought that might

be tasteful to put in the movie Cars. He was a consultant to that show and well, we were all a little surprised. But we had a beer over it afterwards. But Sport Vision has done things we know

that more people have heard of i mean i don't know maybe nascar is the biggest sport in the u.s but football and the yellow line that was the first big sports well it wasn't um okay a group of the the first engineers at sport vision then working for fox sports uh developed the glowing hockey puck. Oh, yeah. You might remember that where the blue and red fuzz ball flew across the ice. And it was blue if it was fast and red if it was super fast. Yeah, yeah. That ended badly.

Yeah, that didn't last, did it? Well, it ran for, oh, three, four years. And once Fox lost the rights for hockey, the next broadcaster didn't pick it up. Because you had to modify the hockey pucks as well as overlaying. How does that even work? Yeah, that was a very complicated system. They tried a bunch of different ways to track a hockey puck. How do you track a hockey puck?

Well, maybe you paint it bright green. Well, that didn't fly very well. So what ultimately happened is they cut the puck in half and embedded a little package of electronics that had some really bright infrared LEDs on it. So, there were, oh, about a dozen LEDs embedded in this puck that would flash 30 times a second. And so, then there were cameras. Flash invisibly. Yeah.

To humanize. Yeah. And then there were cameras that were up in the ceiling of the hockey rink. And they would track it? They would track it. So, they would see this thing flashing out there 30 times a second. And from multiple cameras, they could get a 3D location of where that puck was. But it takes some abuse.

I mean, the hockey puck isn't known for being this fuzzy little ball that gets tossed from player to player. Yeah. Well, it turns out that, first, hockey pucks don't stay on the ice very long. Right, because they put new ones on all the time. They put one on all the time. And it was a packaging problem of how to embed these LEDs in the puck. But it worked quite nicely.

Okay, so how did you embed the LEDs in the yellow line in the football? Well, my favorite demonstration of how that was was when the directors, it was on one of the 49er games. They did a little highlight on how they do this. And so, they got up before the game and had a paint roller and painted a yellow line across. That's the example. And then when he had to move it, they'd get the green paint out. No, seriously, that's done with augmented reality. I like that buzzword.

That is a great buzzword. Yeah, yeah. The instrument, the cameras, so the broadcast cameras had pan and tilt sensors on it. And then we could get inside the lens and measure how the zoom was moving. And then after you calibrate all that, figure out where the camera is and what the zoom measurements mean, then you can bring that video into the computer and the data from the pan tilt head

and compute where the first down line ought to be. And this is also information from the sidelines, right? I mean, because they have the flags. Do you use the cameras to see the flags? No, an operator just with his mouse clicks on the field, the virtual field, and that's where the line is. So it's unofficial, but they're really good at it they are but i remember we were on a business trip with with crossbow and the cars and we it was after work and the question was what were we going to

watch that night and there was some i don't know sci-fi or mystery thing on and i thought you guys all wanted me you were trying to make me happy to watch it and i was like no let's watch the football game i'm i love football it's because you guys couldn't watch football all you could see was the yellow line and it was like oh no it's over the foot oh oh now it's under the foot and you were so fixated can you watch sports no, it's over the foot. Oh, now it's under the foot. And you were so fixated. Can you watch sports at all now? It's just horrible. In those days, you're right.

All we could see was the imperfections. Today, I'm over it. I can watch sports. I love sports. Are there any soccer effects happening? We did a little bit on soccer, but that's really not the sweet spot for uh sport vision so it's um not there the technology that's prominent in soccer now you don't really see it but they have electronics in the the soccer ball to uh see whether there's a goal or not

yes they were making a big deal about that. Yeah. The TV, whether or not it was on or not. There's a company over in Europe that's doing that work. So back to the yellow line, and not when you're fixated on it going wrong. The pan and tilt camera, and you have to know precisely where it is and where it's looking. That's right. Do you add electronics?

Do you work with the camera manufacturers to do that? There's a company that makes the pan tilt head. That's kind of a standard broadcast pan tilt head. And we buy those and modify them, put optical encoders in them. And then the lenses, we can take them apart and we can modify them and put another encoder in

to measure the zoom and the focus. And then that all comes back through, we've got a custom box that has a little embedded data acquisition system and modulates that data onto an audio carrier and sends it down the audio line of the camera. Because usually the audio on sports broadcasts isn't important. That's right.

The audio is coming from effects microphones and talent. So it's a convenient way to to transport that that data and how many cameras on a field will have this modification how many cameras are there on a football field a big football game like the Sunday game might have 15 to 20. Okay. And usually I would instrument just three of them.

It's an interesting thing in a sports broadcast. It's quite formulaic. So the cameras are mounted at specific locations, and then the way the the game is shot there's a pattern to what cameras they use for what part of the game so they'll have a camera they call them camera one two and three camera one would be on the left

20 yard line camera two would be on the 50 yard line and then camera three would be on the right 20 yard line that makes sense but that also explains why sometimes when i look at when they do replays of very old games it's like they're wrong yeah it's because i'm trying to only be able to see from these three spots isn't it yeah and it's maybe it's evolving a little bit and how they shoot it um but so during the the play the actual first not replay but the actual play it's always shot from camera one two or three whichever one's closest to the play and uh you know they'll start a little wide and then play starts and then they'll follow the play and then widen out and then they go to the other cameras maybe an end zone camera or a low camera to shoot uh you know a close shot of the receivers

a and then they might get a shot of the coach yeah so they play it and then as soon as the down is over, they do these other effect cameras. Right. And then sometimes they do the replays. Those sometimes have effects on them. Do they come? Yeah. How does that work? Well, let's say the play is on the 50-yard line.

Okay, well, the camera two will shoot the play. Camera one and three will take different angles to the play well those three cameras have the yellow line on them all the other cameras don't but they don't necessarily they aren't necessarily showing the yellow line uh or if it was during live play they probably would show okay, if it's one of those cameras. If it's maybe when they're moving the line, you know, say on a first down, they would take the line off. Okay. And that's a person, an operator that's making all those decisions. Not the producer?

No, on the yellow line, whether the line is on or or off it would be a sport vision operator that would do that do you have to model the optics of the camera at all or is it just kind of a general okay this is the current focal length and yeah all the parameters have to be modeled uh first there's a process to figure out where the camera is you know you just use an xyz coordinate system that lower or the left near uh goal line would be zero zero zero and then uh in the camera there is zoom and so it gives us a voltage. So that has to be mapped to field of view. And the radial distortion of the lens has to be modeled. You might notice at the edge of a camera, or of a picture in a camera, the lines look curved. That's radial distortion.

And so that has to be modeled because you want to take it out or because it messes with the yellow line that's an interesting you want to match it right yeah because those yard lines will be curved in the camera view because of the lens distortion and so we have to distort the virtual first down line. Otherwise, it would look so wrong. If you saw this very straight line next to a yard line, which is supposed to be straight, but it's not because of the lens. So there's a calibration process that happens before the game to get all those numbers right. How long does it take?

How many sport vision operators are there for a game? There are two. Two. And one of them is monitoring the first down stats yeah well before those start before the game the day before they would set up the camera and perform all these calibration procedures it's maybe a half hour per camera okay that's not so bad and then test it all out make sure it's all working um and uh then during the show there's one person that's deciding when to have the line on and where it should be and then there's another very important part of a virtual yellow line

is that you don't want it to draw over the players yes we're trying to you're trying to create an illusion that this line is on the grass. Yeah, I mean, because you really do want it to look like somebody painted the yellow line on the grass and that it just moves. Yeah. But isn't that a matter of matching the grass tonality? Yeah. Something?

I have no idea. Well, in the biz, it's called chroma key. Ah, yes. Yeah, and so there's a software application where an operator is picking the colors. There's a nice user interface where they can, you know, get the color green that's in the sun, the color green that's in the shade, and the dirt that's on the grass. And even the white that's on it.

Yeah. But not the green and white of the player's uniforms. So you can have inclusion colors that are in the key, the greens, and you can have an exclusion, you know, that special Green Bay Packer green. Yeah, exactly. Yeah. So that tends to take a person person full-time to operate because it changes yeah as the sun changes and everything changes yeah and so these these so that's only two days

a week what do these people do for the rest of the week ah that's that's interesting having met some of these people that question would just get me shot. Yeah. Remind me to take some beer over. When they're not at the bar drinking, do you mean? Exactly. No. No.

Broadcast televisions, it's a funny business area. At a typical football game, let's just say a Fox football game on Sunday afternoon, there might be one Fox employee there. But it takes maybe 60 people to produce the show. And all the people are freelancers.

And so their career is, you know, travel to the show. So for a football show, they might travel on Friday, set up on Saturday. The game is on Sunday. Travel home Monday, and then they'd be off Tuesday, Wednesday, Thursday. That's a tough life. Yes. You get lots of frequent flyer miles.

Imagine. Just trying to collect those. It'd be a good career. And there was always the truck. I mean, do the football, does the Sport Vision operator team have the big truck that they do all of the production from? Or was that just for the race cars? So it varies by sport uh on football the equipment's always installed in one of the broadcasters trucks because it's a small footprint

on the nascar system it's large and they travel a full uh you know 18 wheelers trailer we call them trucks but they're trailers that have power systems and air conditioners and benches and television monitors and computers. I essentially lived in one for a week. Yeah, I did. Okay, you said there were 60 people. I want to talk more about the engineering, but 60 people so there's like the talent and the producers and the camera people and i'm only getting up to like 10 there yeah well if we started in the production area uh takes three people you have a producer he's sort of the big picture guy and a director and he's the person that's uh calling

out what camera so every time you see a um the the camera shot change you know in a broadcast there's a guy back there says ready camera three tick ready camera one fade and then there's a guy sitting next to him that's at a console. It's called a switcher. And it's a big multiplexer. It's lots of inputs and one output. So he pushes the button. Take camera two, pushes the button. So there's three.

Behind them, there'll be another three or four people called associate producers and directors that are helping that team out they might be talking to the studio saying that okay we're going to a commercial in 10 seconds nine eight seven six okay we're out and then everyone chills out for 30 seconds during the commercial and then he'll call them back in. There'd be a tape room. There might be... Because they're doing the...

They're prepping the halftime game as the game is going. Yeah, the halftime show would probably be in New York, in a studio. Oh, yes. Yeah. But they're clipping the interesting plays.

That's right, yeah. So all your replays are coming off tape. And by tape, we do mean digital. We really mean, yeah, digital hard drives. Okay. The old days, it was physical tape. So there might be four or five tape operators that are recording clips, interesting things, and then being ready to play those back for the broadcast. Or, say, send it back to New York for the halftime show.

Like, oh no, so-and-so just got hurt, let's play his interesting down, his play. And then you'd have graphics operators. So, all the graphics that show the names of the player, scores, all that. There'll be four people doing that. There's a person at each camera. Of course. There's a person that's what we call painting the cameras. So they have a... Image of a paintbrush, you know.

Yeah, well, it's not far from that. They can control the color saturation and the color balancing. If you ever watch a... Pay attention at maybe a college show or maybe a show that's... Less produced. Less produced. And you'll notice when they cut cameras, if it's a football game, the green color changes a little bit. Great.

That's bad. Well, that's good. I only really watched the overproduced ones, so that's fine. So that's when someone wasn't doing their job. Because all of those cameras should look identical. And even like at a golf, you ever notice how golf is always so intensely green? No, I don't watch golf. Okay.

But they can make it a little greener than it really is to make it a little prettier. Yeah. Yeah. Painting happens in, it's not a truck and it's not a studio. It happens in the technology area or at the camera? No, that would be a, no, it's not a truck and it's not a studio. It happens in the technology area or at the camera? No, it's in the truck.

There'll be a booth where a guy's got a whole row of controls, one for each camera. And he's just watching the video all the time and tweaking the cameras to make them look so pretty. There'll be an audio person. So he's going to mix the audio. Of course. And somebody getting the audio from, I mean, because they do get audience sound as well as the talent sound.

They get the background sound. Yep. So those will be microphones that are set around the stadium. And then we'd call them the a1 audio number one guy would be mixing all that audio together how have these roles changed over the years it seems like you could i mean go back 20 or 30 years and still have these same same people doing sort of the same jobs. It's just the underlying technology that's changed, which is kind of odd.

You'd think that some of them would be replaced by computers or automation at this point. The painting thing seems like it's rife for automation. It's a bit of an art. Yeah. Well, I've been in this business 15 years, and the workflow has been very constant. The technology certainly has changed. It went from analog video to standard definition digital, and now it's all high definition digital. I've been involved in productions in Europe uh in new zealand in the states and

the flow is the same it just just subtle differences on how a show is produced and does it matter the producer or the broadcaster or is it it just is pretty much the same it's just it's a method and football versus the America's Cup I mean they have different patterns but yeah if you took a lot of the people

and a lot of the jobs you could take someone take them to a football game and take them to baseball take them to sailing and it and take them to baseball take them to sailing and it would be the same the director and producer are more specialty jobs so because they're more in tune with what the audience is looking for yes and they need to know more about the sport so so if you took a baseball producer and plopped him down in a sailing show, it probably would not be good.

So, I brought up the America's Cup because I know you just finished working on that. What were you doing? On the America's Cup, we had a very great opportunity. Larry Ellison, the winner of the America's Cup, is trying to make sailing a more commercially viable sport. He has a passion for sailing, but there aren't very many sailing broadcasts that one would see. So he decided to bring more technology into sailing,

and we developed some augmented reality to paint lines on the water. Okay. Sailing has this, for the sailors in the audience, you know that you can tack upwind, but you can't go straight upwind. And in a sailing race, you might have boats going different directions, yet they may be tied. Because they chose different paths. That's right. You can sail about 45 degrees to the wind.

Okay. But from a helicopter camera or an on-water camera, you just can't see the angles, and it's hard to tell who's ahead. Well, you can't even really see exactly which way the wind is blowing. Right. that from a helicopter we can lay lines down on the water that showed the boundaries of the race course, showed what are called ladder lines. So these would be lines perpendicular to the wind. And if two boats were somewhere on that same line at the same time, they would be even. Oh, neat.

So it just really made the sport more understandable, particularly for the non-sailor. Yeah, that does make it sound more accessible. Yes. And so that, I mean, that sounds like you're drawing lines on something, but I suspect it wasn't very, I mean, some of the technology was similar. What was the same? What was different? Yeah. The big piece of new technology we needed to develop there was to produce the augmented reality from a moving camera.

Right, from the helicopter. From the helicopter. Instead of from a well-understood, located, fixed. Yes. One of three cameras. Okay. And when you have a camera that's mounted and planted onto the ground, it's easy to measure its pan angle. From the helicopter, it's moving around and knowing where north is is a bit more of a challenge. Yeah.

You need a really good IMU to do that. So we found a really good one. And fortunately, we were able to buy it, to use it. And GPS? Yes, we had GPS. Did you instrument the boats too? Yeah, every boat had carrier phase GPS. So we knew where it was to about an inch.

We knew where the helicopter was to about an inch. And we knew all of the angles of the camera from the helicopter to about a hundredth of a degree. That is a nice IMU you have. It was really, really good. Yeah. It was from a French company. Oh, I'm not going to talk about other people's IMUs. Yeah.

This one cost probably a little more than the ones that you were playing with. Oh, yeah. Far more than what Crossbow cost then, even. And far more than MEMSIC costs now. This was about a $150,000 instrument. Yeah. And it was good enough to export it to other countries.

We had to get special permission from the Commerce Department. Well, because IMUs aren't always used for finding helicopters. Yeah, it would fit in a missile, I think, if someone was so inclined. So what kind of engineering do you have to do when you're there? I mean, a lot of the visual stuff, it seems like you can do that in a lab, do that on your desk. But I know from having worked with you before that a lot of engineering gets done on the ground. How do you maintain good engineering in… Yeah. Part of it, it's when you're in the lab it's hard to simulate everything you don't have a big football field um and project you worked with us on nascar we don't have race cars

well and one of the problems with that was that in all of our modeling we were never going fast enough yeah so so you do your best you can in the lab but ultimately you've got to go out to the field and uh there's surprises you know the old motto it doesn't work until it's tested and it's been dunked a few times. So, it was very typical that we'd bring science projects out to the field and the engineering team would travel with the project for four or five shows before things were working smoothly enough to be able to turn it over to the field operators. And how big was the team for the America's Cup for the effects you were saying? Yeah, we had a team of about 10 people. So there were three software engineers. We had a team that would install equipment on the boats.

And these boats are very precisely engineered. So installing things is you really have to do it where they want and you have to work out all the rules. Yeah. Fortunately, in this project, we were part of the America's Cup.

We were employees of the event authority, and we were able to work into the rule that this equipment had to be carried. We did a nice packaging job so it fit into their boats. They had to accommodate the space. We told them we need so many square feet of space. And they just had to do it. If you want to be here, you have to play by our rules. Yeah, yeah.

And I personally had a very hard time doing that sort of engineering on the run. I'm definitely more of a pampered lab rat. But you have had some really great teams. What makes people function under those conditions? Oh, boy. I mean, that wasn't to say I didn't enjoy our lovely trip to Florida where I saw nothing but the inside of your van. You know, for me, maybe I'm a bit of an adrenaline junkie,

but being there on the spot is exciting. Having to get it done before showtime because when showtime happens, that's the only part that matters. Yeah, it's different than a lot of jobs. You know, Microsoft could delay Windows for a year. Certainly for an hour is not going to make a difference.

Yeah, but for the Daytona 500, you know, on February 3rd at 12 o'clock, the race was going to happen. And you couldn't go and say, well, could we run it tomorrow? The engineers just need a couple more hours to fix these bugs. Yeah, no. And so, if you don't finish it, the consequences is bad. You don't get paid for the show, and it's embarrassing.

So, I think the people that excel in that, they just like the pressure. And a lot of them loved the sports. It's a mix there. An awful lot of our engineers had no clue about sports. It was funny. It's very shocking. An engineer not liking sports?

Yeah. It was a real mix. I mean, on NASCAR, I hardly knew what NASCAR was about. I ultimately became a fan. It was, you know, people say it's not a sport because they're just driving a car. Now, after one of those tests where they let me ride in a NASCAR going 180 miles an hour, that was hard. Two minutes later, I was exhausted.

Yeah. Yeah. They are athletes. And then they had to think under those conditions, think hard. Yeah, I was impressed by NASCAR after having been around. Are you a fan? No, I don't watch NASCAR. Oh, well.

You'll be perhaps entertained that a lot of the drivers from from that year what was it 2000 year 2000 that we were working on that so it's uh 14 years later a lot of the same guys are there i think they can run that race when they're 50. i i don't know how man it was so was so hot. It was so loud. Isn't that horrible? Football games aren't loud like NASCAR. Yeah. Not NASCAR. Are there any sports that are that just draining from the volume?

I think motorsports has to be the loudest. And you did motorcycles, too. Well, we did some testing on motorcycles. Okay. And you did some Formula One stuff. We did some testing on Formula One. Which of the motorsports is the best? Oh, well, that's just an opinion.

If you're an American, NASCAR. If you're a European, Formula One. If you're a bike rider, motorcycles. You're not going to come down with a Ken's choice, is this? 49er football? It's not a motorsport. Or if it is, I've been watching the wrong games. Just make robots.

Yeah. Where do you see sports technology going? Well, a lot of the things that we've done at Sport Vision are getting easier to do. Computing horsepower is so much higher. Yeah, because in year 2000, I mean, you still had SGI, like O2s? O2s, yeah. Some huge, monstrously expensive computer. At the time, in 2000, an O2 was a very powerful desktop computer. We could just barely render 30 frames of video in real time.

Had to just be really careful with the code. Today, a PC can do high-definition video, 60 frames a second, and be cool. So that part's getting easier. I think the quality of the rendering will get better and better, more realistic. I think you can see that in broadcast. I think there'll be more tracking, tracking of balls. We're already tracking pitches in all the Major League Baseball, but the players will be tracked, football players will be tracked, and then that's going to generate a lot of ancillary data.

The statistics and whatnot that are then used to build the video games. That's right. And if you go to your Major League Baseball app, you see where all the pitches cross the plate. Well, that's all track data. It's real. And you'll soon see that there'll be the players tracked. How fast they run? Sure, all those things. Whether they hit first base on the left side or the right side?

Yeah. So does all that data go back to the teams so they can do it it does uh on baseball the teams get the data it's actually publicly available there's some secret servers that uh the cyber meticians use to used to get the data. NBA players are currently being tracked. That data is not public, but the teams get it. Are they being tracked and they're going to get little accelerometers to find out when they fall down if they were actually touched first?

Sorry, basketball's not. Yeah, well, you know, that's conceivable. I think that's a little out there. That'd be fun with soccer, too. Yeah, right now they're tracking the players with cameras and, you know, they're tracking the blob of the player. Yeah, center of mass. Yeah.

But seeing how hard these players get hit would be interesting. Yeah. I'm sure they want to know. It's true. I'm not sure I really want to know. Yeah, yeah. There was a little bit of that going on with IndyCar. I think they had accelerometers in their earplugs. And they did that really just to understand the impact

and how to make the car safer. That's good. Yeah. So there'll be more of that. So traveling a lot. I mean, you've traveled a lot. Yes. Do you have advice to people who are thinking about such a nomadic lifestyle?

I mean, you'd be gone for five days every week for months. Yeah. Well, maybe some of the lessons learned, you know, you got to keep your family life together. Yeah. There were a lot of single operators. It's not uncommon to see marriages broken up for some of the road people because you're just not with your spouse or family. It's a difficult thing. If you wanted to become a field television person, you've got to like travel.

You've got to do it. Because that is the life. You build a new family out on the road. Because you often travel with the same people. So I think that's important. Or get off the road. That's what I did. I'm not traveling so much anymore.

You've said that. But you've said that a few times. And then you did the America's Cat. So I'm not traveling so much anymore but you've said that a few times and then you did the america's cat so i'm not sure i believe you anymore so you have good stories from traveling i did hear something about a nobel prize winner that i wanted to know more about that was that was an amazing trip we we were over in france uh at a formula one race and we made late travel reservation so we had to stay at a uh an inn i think they called it a castle it didn't look like a castle to me but out in rural france those europeans what do they know what castles look like? Yeah. So we checked in, and the guy comes and runs sort of like a bed and breakfast. But there was a funny thing about this guy.

He just didn't seem like an innkeeper. It was almost like the innkeeper was gone, and it was the neighbor, and he says, okay, well, here's the key. There's your room. So we got there, and we tried to get the Wi-Fi hooked up and it wasn't working. And this is a problem for engineers. I mean, you have a bunch of software engineers with you.

Yeah, we need Wi-Fi. And Wi-Fi is just like more important than, well, it's more important than bathrooms, but slightly less important than beer. So we went down and talked to this guy, and in our English and his French, we finally got the message across that the Wi-Fi wasn't working. And so, he took us in tow into his office and said, well, there it is. You know, fix it. And we didn't get very far on that we didn't know the password but and and going

into his office we noticed that there was a plaque on the wall it was a nobel prize so we got to talking to this guy and it turns out he's a nobel prize winner because that's what you do after you win the Nobel Prize Not in networking apparently No So he was a He worked for the UN And was on a negotiating team That negotiated on one of the civil wars in Africa He won a Nobel Peace Prize Yes

It was just this amazing thing Wow And reflection We were right that this guy wasn't in the right spot he didn't belong as an innkeeper he was an omelette price winner for for peace negotiations so that was a lot of fun are there other i just tripped upon this amazing thing, sort of stories that you've gathered. Oh, gosh. It's been fun, you know, particularly in NASCAR, that we had to work in the garages.

We had to put equipment in the cars. That's how I ended up upside down in a car in the pits an hour or two before race was about to start. That's always a good story for me. You just get access to things that you wouldn't otherwise. Yes. Oh, yes. I mean, you get backstage access to all these things. Yeah.

I mean, you get the badge that lets you go wherever you want because the chances are you need to go wherever you're going. And yet, you're working. Yes. Is that hard? I mean, do you get to take time to play? You make time. You make time.

Yeah. Like in the garage on NASCAR, you know. Become friends with the crew people, and they're friendly people just like most people are. So, become friends with them. And sometimes there are failures during TV broadcasts. And sometimes the audience notices them. But for the most part, I don't think we see the problems that happen.

How do you cover up the failures? How do they cover up the failures? Yeah. It's a good thing to highlight. In a broadcast, you have 20 cameras, and you're trying to tell a story. You have graphics and the toys like a yellow line. But, you know, if the yellow line's not working, well, you just don't put it on air.

Or if it's jumping a little bit for some reason, you just don't put it on air. So there's a lot of things that are covered up. Hopefully not very many, but it happens. And most people won't notice. I've seen times, well, this big multiplexer I spoke of,

the switcher. It's driven by a computer, like everything else. And there was one show where it crashed during the middle of the show. Does that mean you were stuck on one camera? Well, for a little moment it was, well, actually went to black. There was no picture.

That's bad. But they have some backup systems to that that you aren't able to fade between cameras or blend in graphics. But you can switch the show. So I was at home watching a show and I could tell that happened. Because I noticed all the graphics were gone and there were no fades, it was all cut. But I doubt many people noticed. But I knew back at the TV compound, it was chaos. People were shouting and trying to get this thing working again.

Yet at home, football game's still on. It's great. We haven't gotten to commercial. Yeah. It wasn't like the Super Bowl where the power went out during the middle of the episode. And generally,

the people, the engineers and technicians, they're pretty calm about it. It's a characteristic of the people out there that they know how to recover from these things quickly. Well, that's the high pressure. That's enjoying the high pressure and being able to work through it. Yeah. Getting the adrenaline rush and taking care of what's going on.

Yep. That part I could all identify with. I think it was the travel that killed me. Yeah. I don't travel well. Do you travel for pleasure now? A little bit. You do?

I probably take a trip or two a year. That's not so bad. Yeah. Having traveled all around the world with this, do you have favorite places? You know, I always play a little bit of a game, you know, would I like to live here? Yeah. When I'm at a, somewhere in, well, in the States, you know, the city that I've always enjoyed being at was Charlotte, North Carolina. It's a pretty city and

nice folks. The other place I really enjoyed was Venice in Italy. We stayed on one of the outer islands called the Lido. And it was just a wonderful little island. It was where people

lived. On the Venice tourist island, people don't live there. But they live out on the Lido and it was just calm and beautiful and very Italian. Until the race cars got there. Yeah. Then it got a little louder. Yeah, no, we had sailboats there.

Oh, okay. Yeah. They weren't so noisy. No. What are you doing now? Well, the America's Cup ended last October. So I've been doing some consulting back to America's Cup and Sport Vision. Oh, I'm doing some of my own little projects.

Did I hear about a chicken coop? Oh, a silly chicken coop. Well, we've got some backyard chickens, a couple eggs a day. And chickens are relatively low maintenance, but you have to let them in and out of the coop twice a day, or, you know, out during the day and at night. So when we do travel or go away for a weekend, you've got to lean on a neighbor or someone to let the chickens in and out. So I put together at the tech shop a little mechanism with an Arduino to raise the chicken door up in the morning and let it out at night

at night what do you use for an actuator um i devised a screw drive so a half inch threaded rod that will is driven on a motor um and it pulls a string up to kind of a guillotine door. And then I have an Arduino with a photo sensor. Yeah, that makes sense. Yeah. Battery-powered, or did you run— I ran power out there. Yeah, because the motor batteries little.

Yeah. I guess you could have a solar panel or something. Rev, too. Yeah. Well, I think we are about out of time. Are there any thoughts you'd like to leave us with? Well, I thank you, Alicia, for this opportunity to talk about my broadcast career. It's a fun career, and uh hope everyone enjoys the yellow line and

they enjoyed the america's cup well thank you for coming it's been really interesting to hear about the different broadcast things usually when i see a show i just think about the show and not all the work that goes into it. Well, Christopher, you have any questions? No, it's amazing to see how much technology is behind things that are purely entertainment. I think people forget how much engineering there actually is in places where you just kind of miss it. Yeah.

Of course, now the next time we're watching soccer and they don't change camera angles, I'm going to be like, oh, I wonder if something went wrong. And thank you for listening. If you have questions for Ken, I'm happy to pass along notes. Hit the contact link at embedded.fm or email us, show at embedded.fm. Thank you to producer Christopher for being you know, being in the microphone. He didn't talk much, but he will more next time, I think. And especially thank you to Ken.

I have a note here. Oh, look, I have a job request. People looking for good software engineers. Let's see. I have a friend at Park's Computer Science Laboratory looking for a few great software engineers. This is a thinky lab environment, not the crazy startups I usually talk about here. Though they're shifting from pure research focus to more commercial, probably by solving hard problems than spinning out startups from there. They've had some spin-outs you may have heard of, Juniper Networks, Adobe, 3Com, Sun Microsystems. Even Ethernet and Apple's original Macintosh owe a lot to Park. It is a neat place to work, smart people, and great work environment and lovely scenery in the hills of Palo Alto. They're building the next next generation of high performance routers too. So if you have hands on experience in Linux and you have some networking experience, they want to talk to you.

They also would like you to be familiar with building virtualized device interfaces, hypervisors, Quagga routing suite and software defined networking code. I have no idea what Quagga is, but if you do, or if you've got everything but that, and you're interested, send me an email, show at embedded.fm, and I will send you the rec and connect you with the people who are involved. This is a job in Palo Alto, so those of you in the wilds of South Africa, Australia, and Ohio, I'll try to find something global next time. And now, final thought. Final thought this week from Michael Jordan, which seems sort of appropriate. I've missed more than 9,000 shots in my career. I've lost almost 300 games. 26 times I've been trusted to take the game-winning shot and missed.

I've failed over and over and over again in my life and that's why I succeed.