Embedded - 162: I Am a Boomerang Enthusiast (Repeat)

Episode Date: March 1, 2019

Valve's Alan Yates (@vk2zay) spoke with us about the science and technology of virtual reality.  Elecia looked at the iFixIt Teardown of the HTC Vive system as she was unwilling to take apart Christopher's system.  Alan shared some of his other favorite reverse engineering efforts: Doc OK’s Lighthouse videos, documentation on github by nairol, and a blog by Trammell Hudson. Alan's sensor circuit diagrams were on twitter: SparkleTree sensor circuit (think simplified) and the closer-to-production Lighthouse sensor. Make Magazine talked about Valve's R&D Lab. This is important in case you want to work at Valve (they are currently hiring for EE but if that doesn't describe you and you want to work there, apply anyway). Alan also has a website (vk2zay.net) though it doesn't see much updating right now.  

You are listening to Embedded. I'm Alicia White. My co-host is Christopher White. We have finally reached virtual reality week. Yay! In fact, Valve's Alan Yates is going to be speaking with us about building the Vive VR system. Now, last week when I interviewed Kat Scott about computer vision, I read most of the books she co-authored. This week, I'm really glad to be talking about VR because all I had to do is spend several hours playing games. And if you don't have your own VR system and want homework before listening to the show, iFixit did a tear down of the Vive system, showing what electronics are there and how they're connected.

Just so you know, we're going to get to some details here. Hi, Ellen. It's great to have you here today. Hello, Chris. It's great to be here. And you too, Alicia. Could you tell us about yourself? Sure. Well, I've been at Valve now for about four and a half years. I'm a hardware slash software slash anything goes kind of engineer. I've been working primarily in the VR group, and I'm responsible largely for the tracking system on the Vive

and many other subsystems as well. Before Valve, I used to work as a network hardware person, and then before that, I used to do web stuff, like websites, you know, Terrascale kind of web. And long before that, going right back, I used to be an internet service provider right out of uni. I actually left uni to do that for many years until it became kind of unprofitable and boring.

Wow, that's quite a diverse set of things. That's great. Yeah, I've been a lot of things over the years. It's, uh, it's been quite a journey to, to get here and obviously the journey continues. So we're going to go straight to lightning round where we ask you questions and want short answers. And then if we are behaving, we won't ask you for follow-up, but that never works. Okay. Sounds fun. Chris, you want to start? Sure.

Favorite electrical component? BJT. Pretend I'm a software engineer. Plain old-fashioned bipolar transistor. Oh, right. Okay. Least favorite? Least favorite.

Ooh. I got to say Xenodiode. Okay. What is your favorite spacecraft? Ooh, favorite spacecraft. I've always liked the Mercury capsules. Yeah. Because you could understand the whole thing. I seem a little cramped. Well. Oh, yeah. Very cramped. But yeah, definitely built at a time when you could understand everything that was going on in them. What is the most important tool for your job,

a whiteboard, soldering iron, or keyboard and mouse? Ooh, depends on which particular job I'm doing that day, which hat I'm wearing. Probably, I'm going to say soldering iron. 8 or 32-bit? Ooh. 32-bit? Ooh. 32-bit. Favorite wave, ocean, sound, or electromagnetic? What? Let's go with electromagnetic.

Most exciting science fiction concept that is likely to become a reality in our lifetimes, not including virtual reality. Ooh. Mind uploading. Terrifying. Yeah. Oh, yeah. Let's go to a favorite fictional robot after that and see if they're connected. I'm thinking Twiki.

Twiki. Bidi Bidi Exactly I used to love that When I was a kid Speaking of being a kid What did you want to be When you grew up? Ooh

Now that's a I actually wanted For a long time To be a doctor But I ended up Well I ended up Being a software engineer But you know

Eventually ended up In electrical So I wanted to be a lot of things. Favorite physical constant? I was going to say the gyromagnetic ratio, but you know what? I think the fine structure constant. All right, cool. Science, technology, engineering, or math?

All of the above. If I have to pick one, engineering, or math? All of the above. If I have to pick one, engineering. Okay. And this one, Chris tells me, is a little too obscure, but I think it has to do with the show. And so I really want to have an honest answer here. Okay. Phil King or Jeff Kaiser? Oh, no.

Okay, you got to edit this out, all right? This show will never air. It depends on what I want done, I guess. How diplomatic. Diplomat. Is that good enough? Because I can just see them on Monday coming in and killing me. Oh, no, it would be Wednesday. This doesn't air until Tuesday.

You have days to live yet. Days and days. Okay. What is a workday like for you typically? For me? Ooh, it can vary. I'm not a morning person, so I tend to come in a little later and then I'll hopefully won't have too many meetings and I can actually get some work done. Lately, we've been obviously working with a lot of partners, and that part of my day is not always fun.

But then there's try and get through some email, do some whatever the problem of the day may be, and that can be incredibly varied. That can be something that's completely outside my domain that I've never even thought of before, or it can be something incredibly mundane like calculate this dropper resistor for this LED. Or it could be hiring new people, or it could be assembling some furniture that's just been delivered. Like yesterday, I had to assemble a chest of drawers that I want to put all components in so that I can do some rapid prototyping. It can be just about anything. I like that. I like

it when my job is that varied. Does it still seem like startup life? You mentioned putting together furniture and that's something that to me was like, oh, that's what we did at startups because there was nobody else to do it. Absolutely. Valve is very much like that. Everyone does everything. There's no real support people. I mean, we have, you know, obviously people that do, you know, HR and have more dedicated tasks, but most of us, whatever has to be done, someone has to do it. It's, you know, not glamorous, you could say,

but it's very equalizing, I think. Everyone has to do the hard work as well as, you know, the good stuff. And sometimes when you have a hard technical problem, building a chest of drawers is exactly the right thing to get it done. Oh, yes. It can take your mind off things. I know when I actually, yesterday, when I was putting together that thing, I came up with a couple of ideas for some high-frequency sensor stuff

that I want to play with next week. Okay. for some high-frequency sensor stuff that I want to play with next week. Okay, so the Vive VR system, which isn't marked Valve, it's marked HTC, but I don't even want to talk about all of that. I just want to talk about the system. Tell us how it works, as though somebody had never seen it. Give us an introduction to the system for complete newbies. And we'll get into more detail because Chris and I have one and I have way more detailed questions. Okay. I guess

for someone that's never seen it before, it's the closest thing we have to a holodeck is probably how I would put it. The room doesn't magically change around you. I mean, obviously it's a display device that you put on your face. But then through that, it has a good tracking system so that it knows precisely where your head is. And you have two controllers that you hold, so you know exactly where your hands are. And really, it opens up almost an unlimited number of experiences that content people can make to take you other places. That's kind of how I would describe it. Well, and the other places is a nice thing to emphasize because one of the technology demos is you're underwater, just sort of sitting on this pirate ship and a whale comes by.

And when you look at the comparison of your size versus the whales, it's correct. But you don't really think about how huge whales are until there's one standing next to you. Swimming next to you? Yeah, yeah. Being next to you. Absolutely. I didn't really have an appreciation for how large. I think that's a blue whale. They really are until, I mean, I've seen their skeletons, but to see one like swim up alongside you, it's like, wow.

It's such a different place. Like, I'm not much of a scuba diver. I don't know that I'd ever had that experience any other way. Yeah, and that's just a tech demo that you don't even interact with. There are a couple others in that same True Blue set that you can play with the fish that are swimming by, but it isn't really much. Yeah. you can play with the fish that are swimming by but it isn't really much uh yeah but when we when we have like um what is the space pirate game is that what it's called and you shoot at little balls yeah space pirate trainer um so that one not only are you in a world and experiencing that somewhat scary world as things shoot at you, you are also interacting back, and that's the controllers.

Yeah, the controllers really are what makes the system. I mean, the display is amazing and the tracking and everything else. There's a whole lot of really cool tech there. But getting your hands in it really is what makes it something else. Otherwise, as you said, it's a fairly passive experience. You can sit there, you can look around. Maybe you have like a controller and you can move yourself around in the world, although that will cause nausea. We know that quite reliably.

Having your own action in the world, your own freedom to move about and touch things, interact with them, is really what gives VR its power to take you to some other completely different experience that you've never had before. And to cause you to experience extreme muscle soreness in muscles you've never experienced before. Yes, absolutely. There's a lot of people. I read the Reddit forums, obviously, and there's a whole bunch of people saying, oh, my God, my thighs.

They've got their leg workout in only a couple of minutes. Well, I don't think people who've never tried it understand that part, just how exhausting it is to hold up the controller as you're, you know, playing lightsabers or crouching a lot, you know, moving from the crouch position. And I do this thing where there's balls that come at you and you have to block them with shields and it's to music and it's like punching and dancing at the same time. It's exhausting.

Oh, yeah, that's a fun workout, isn't it? I love that one. So what were the most difficult and foreseeable challenges with building a system like this? I think getting everything right. Throughout the years, right back from before the 90s, first kind of VR, consumer VR came about, everyone's had kind of bits and pieces of the problem. Display technology took a while to mature and now obviously with MEMS and other technologies, we've got much better tracking than we ever had before. But getting everything like display technology took a while to mature and now obviously with MEMS and other technologies

we've got much better tracking than we ever had before. But getting everything sort of together and getting the optics right and getting the tracking right, getting the latency low enough, and then also GPU power being sufficient to actually render sort of photorealistic worlds, obviously all that's going to improve in the future, but it's now at a point where it's good enough that it can convince your brain, your neural system that you are actually in a different world, in a world that can range from cartoony, which is surprisingly compatible.

Like people like Job Simulator, for example, that the world's relatively cartoony, but it does seem very real to you when you're in it, to the almost photorealistic things like the blue. And have people talked to you about having this world, the real world, be weird after VR? Yes. We don't have a name for that yet, but there does seem to be a post-VR kind of thing when you come out of it, particularly people who've only, you know, never used it before, basically. The first time they come out, you see their eyes are wide open and they're like, wow, mind blown. And for some people, I guess it's like the Tetris effect, you know, they kind of experience an after effect for a little while. We did know from research into the vestibular system that people, we're actually stimulating the vestibular system that people, we're actually stimulating the vestibular system more or less correctly because your motion is one-to-one in the world, but the optics aren't

exactly how the real world behaves. So there is kind of an effect there where people develop a second slot, if you like, where they kind of interact with virtual reality a little bit differently than reality. And the vestibular system is what's responsible for your equilibrium. That's like your inner ear and the way your body is, your proprioception, how you feel where things are. Yes, absolutely. Yeah. It's probably the most difficult sense to, well,

it's the one that will make you sick if you get it wrong. Let's put it that way. So did you find, did you have in the back of your mind when you guys started this kind of thing that you'd have to do kind of medical research or at least a little bit of physiological research to understand how this affects the body? Were there new things that you kind of came across that you didn't expect? Yeah, we went through many of the old papers. Obviously, the military had done a lot of experiments because they've wanted to do simulations for training soldiers for a long, long time. And there's much research out there. Unfortunately, a lot of it was done

at a time when latencies were much higher and the general fidelity of the systems was very poor. And as a result, much of that research, unfortunately, is kind of tainted by that. A lot of it's actually quite valid though. Much of the stuff about simulator sickness was certainly borne out by our experiments. We had experiments where we tried to acclimatize people to undirected motion, where they're either using a keyboard or a mouse and a stick. And that makes most people quite ill. And we would expose people to that for a little period of time and then would back off and then once a day they'd come in and play until they sort of felt unwell and then they'd take it off

immediately and not do anything else for the rest of the day. And by doing that, they could build up some tolerance to it, but some percentage of the candidates, the test people, just couldn't do it. They could never develop VR legs. So that's where we decided that we just shouldn't do it that way. And we should give people that autonomy in the world to move around and for the motion to be one-to-one so that we just sort of completely bypass that effect of causing nausea from emotional vection on their visual field.

I don't want to ask how your test area was set up. Plastic in buckets. No one actually got quite that sick, but there was definitely a lot of ginger consumed after some of those early experiments. So did you, through those experiments, come to a, it has to be this fast, it has to be this accurate specification stage?

Yes, we got a pretty good idea of the latency that we needed and the accuracy of tracking that we needed. And that's kind of what drove a lot of our development after that. It did give us a reasonable idea of what display quality was required. And unfortunately, as far as displays go, there's really even the current ones and any of the foreseeable ones for the next couple of years are not quite good enough. But we know now where we need to be. That was going to be one of my questions is when you got those numbers and you looked at them and you thought about them, did you realize, holy crap, there's no way we can do this?

Pretty much, yeah. I mean, the latency and things like that, I thought, okay, obviously we can do that. That's just a matter of engineering. It's be the light, be damned kind of thing. But when I looked at the numbers and looked at where GPU performance was, it's like, okay, for many years, GPUs had kind of stalled out and it was getting difficult for the GPU manufacturers to sell them because people didn't need any more than 90 hertz or whatever they could run at. And now we've given them reason to have many orders of magnitude improvement.

Which is happening pretty much this year. I mean, the cards that have come out in the last month or so are just starting to be, okay, this is really capable for doing this. Absolutely, yeah. We decided that 90 hertz was kind of the minimum that you'd need, and you have to render in stereo however many pixels you have. So two times that at 90 hertz, and you can't drop a frame. So that's a pretty high bar,

although, yep the the silicon guys have certainly stepped up to it and they're obviously going to enjoy the the new uh tail of that as it uh as technology improves so motion sickness seems like one of those things that you knew you had to to battle and getting the gpus fast enough to be able to handle the latency is another area. Did you ever sit down and think, well, how are we going to put this person in a room? How are we going to track their head? Absolutely. Yeah. Tracking has always been the difficult problem in many ways. I mean, display is interesting, but you could see that it's, you know, Moore's law is going to help you there. There wasn't anything particularly difficult about it apart from just getting that many gigabits to the headset. In terms of tracking though,

that was one of those things where, well, we need sub-millimeter, you know, kind of milliradian pointing error kind of tracking over whatever volume we're going to put the person within. And for a long time, it seemed like that would be an expensive problem or we could only do it to a certain amount of fidelity. And that's where kind of Lighthouse and other tracking systems like it came from. There was actually many other tracking systems that we developed here. When I sort of first arrived at Valve, I sort of took on the tracking problem and I went through all of the physical ways that you can possibly track the position and orientation of something. You know, in this universe,

there's only so many different ways you can do it. You can either have it emit some radiation or you can sort of throw some radiation at it and you can get a shadow or a reflection. That or, you know, matter or whatever. I mean, you could potentially do it with x-rays or a whole bunch of other different things but you don't want to like fry the person in the room that's usually not found upon yeah there's some physical limits on what's possible um many of the early tracking systems like the the sort of damocles system were essentially mechanical they had you know tendons on coming off onto big mechanical things that hung the system from the ceiling. And that obviously was super low latency, but there was resistance there and very, very expensive.

You had to instrument the environment. One of our tracking systems, the only one that you've probably seen, is with the fiducial markers on the wall. That system is actually very good and simple to implement. You've got one camera, although there are some technical challenges associated with it making it work well, it's not very deployable. It's not something you can ask people to paper their walls with these, you know, fiducial markers, although I'm sure some percentage of the people out there

would totally do that. Yeah, it's one of those things where we needed a much more fieldable solution. So we developed a couple of different tracking techs and Lighthouse is kind of the last in a long line and probably not the last kind of tracking system we will develop. Okay, I want to ask about Lighthouse in a second, but why couldn't you just use MEMS sensors? Right, so MEMS sensors are, you know, accelerometer and gyro, they give you angular rates, so they give you accelerations. And then you have to, to get position, you obviously have to double the integrate acceleration. So integrate once to get velocity and then you integrate again and all the noise adds up and basically you can't, they drift, right? They have biases and the noise just kills you. If you integrate, free integrate an IMU for more than a couple

of hundred milliseconds, it will literally fly out of the room. Yep. Yep, yep. Sorry, I love inertial sensors and trying to explain why they're never quite good enough for what you need is a hobby. Absolutely, yeah. I'm currently, one of my little projects, I'm a boomerang enthusiast and I've been trying to track and instrument a boomerang in flight and inertial sensors are good but they're not good enough. So I'm probably going to put optical tracking on a boomerang at some point. And GPS isn't fast enough, so you can't do that.

Unless you get a really expensive IMU, you can't integrate through GPS. Which you're not going to be able to put on a boomerang, probably. Right, no. Yeah, absolutely. The lighthouse system, which is what Vive uses, is sort of the opposite of what Oculus Rift uses. Can you compare the two technologies for positioning people? Sure. Yeah, they are kind of mathematical duels, but they're not quite.

There's a couple of subtleties in the differences between them. The Oculus, let's start with that one, and similar tracking systems. There's been many blinking LED or synchronized LED and camera, either pixelated camera arrays or linear camera arrays. So there's some commercial mocap systems that use linear cameras, but they're outside looking in essentially. They have camera sensors that have angle of arrival measurement, basically, of some kind of emitter, in this case, LEDs, in the volume. And some constellation of those, that's probably why they call it constellation, some configuration of those points is rigid. And the cameras can see it from different angles and basically do a pose end points kind of solve inversion of the perspective projection that the cameras

see of those LEDs. Lighthouse kind of works the other way around. Before we go there, I just want to clarify that with the Oculus, there are LEDs on that headset and there are cameras around the room correct yes i just wanted i i know you said that but it was it was a little tough and and that ends up being a time difference of arrival problem if you have multiple cameras as well as there's the fixed body and so you can kind of see where the person is because they've got this LED here and that LED there and this other LED here and it arrived at this camera then

and it arrived at that camera there. And so you can calculate where the person is faced and where they are in the room based on the LEDs they're emitting in a particular pattern. Yeah, yeah. It's not so much time of arrival as angle of arrival. Basically, a lens turns angle into spatial position on the sensor. So there's some mapping, kind of like a tangent-style mapping between which pixel the image of that LED appears to arrive at

into angle space. So then you know where the cameras are, or you can work out where the cameras are relative to each other if you have more than one. I mean, their basic kit is very similar to ours. It'll track with a single camera or we can track with a single base station because you have multiple points.

So that orientation as well as position of that object makes a pattern on the sensor that you can then fit a position in three spaces you know a six-dimensional um number basically that says where that object is cool okay now lighthouse works the other way it puts lights on the outside and the and the detectors on the on the headset right yeah correct yeah so it's kind of the other way around. So it is in a sense, the mathematical dual, you could call it like the phase conjugate of the propagation of the radiation, but it's a little bit more complicated than that. It has, obviously there's some

disadvantages to doing that. You have to put sensors now rather than just LEDs on the tracked object. And that's, it makes the tracked object a little bit more complicated. But it has other advantages in that, for example, cameras have been built using glass and silicon, that they have finite field of view and finite depth of field. You're kind of limited by what you can do with optics and that tends to scale in price quite aggressively. And also, if you have multiple objects in the room, the camera is kind of where all of

that data is being received and if you want to do um you know multi-person interactions where you have multiple multiple trapped objects in the volume you have to then get all that information out of your tracking system and send it somewhere else so you either have to send it to the computers that are doing the rendering or to the tracked objects themselves if they want to be fully autonomous. So Lighthouse kind of inverts that problem and makes the complex part, the solving part of the problem, the responsibility of the tracked object itself. So it's more like GPS. You have these, we call them base stations. They emit essentially a structured time-based light that they output. And by looking out at the field of lights emitted by one or more base stations, you as the tracked object,

you know where your sensors are and you see these pulses coming in at different times and you can work out where you are. So in many ways, it's actually quite similar to GPS. The complexity is mostly in the receiver and the receiver has the ability to autonomously work out where it is. So it scales very well. You can have, you know, as many objects in the volume as you can put in there essentially, but they are a little bit more complicated. And when I'm playing my space pirate trainer game, I have three tracked objects. I have my head, which is also providing display for me, but that's not important for tracking.

And I have the two controllers, which are acting as my guns or shields in the game. Is that right? I have three tracked objects, or does it not work that way? Yeah, that is three tracked objects. Each one of those has its own independent, we call it a watchman device, which is kind of the receiver that takes all those signals and correlates it. At the moment, it sends it back to your PC because that's where all the rendering is happening. The advantage is if you want to backpack a computer or have a completely autonomous mobile device that was just, for example, built into the headset, it could do everything that

it needed to do locally. It wouldn't have to go via a lead or radio or whatever to cameras that were mounted on the you know the edges of the volume right if it was the other way as with oculus then the cameras have the tracked objects information and so they have to join up somewhere. And having the detectors on the headset means that ideally when computers are small and even faster and more power efficient, this all can be on my back or on my head. Correct. Yeah. So that's one of its primary advantages in terms of scalability. The cameras sort of suffer from, you know, kind of Alba's paradox kind of thing, where why is the

sky not bright at night kind of problem. If there's a bunch of things in the room that are all trying to be tracked by cameras, they all have to run different codes. And obviously you have to deal with code space or you have to track them in angle space. All of these problems are certainly solvable, but some of them you sort of just get for free if you turn the system inside out. There's also advantages in depth of field. The lasers are, you know, they have very low divergence, so you essentially have infinite depth of field.

And we can, because of various ways about the system, the way the system works, we can get pretty good angular resolution, probably better than what you can get with an equivalent price camera, just by virtue of the fact that we have these low divergence things and we have very accurate spinning of the motors in the base stations. Okay. Now, this we need to talk about more because we've been kind of saying cameras and detectors

and LEDs, but what it actually is, is the headset has a whole bunch of detectors, photo diodes. And then these lighthouse things that you've made put out, you said structured information, but I was told it was like a grid. And there are two in the room sort of at opposite ends, sort of diagonal is how we did it. And they put out slightly different grids. Is that right? Yeah. The system that you're using at the moment is one of many different ways you can set up Lighthouse. So you can actually track with a single base station.

But when you turn your back on a base station, for example, you would shield the controllers. I mean, like all optical systems, it depends on line of sight. Like I was joking about before, we could use x-rays or some other kind of penetrating radiation and go right through bodies and including objects, but would also probably cause a lot of cancer in the process. Neutrinos are fine. Yeah, neutrinos are fine. The detectors might be a bit challenging. So we have one or more base stations to cover the volume. Now what they emit, there's a synchronization flash that's sort of an omnidirectional, well not omnidirectional, but covers 120 by

120 kind of degrees pulse of light. It's a little bit more complicated than that. There's some modulation involved, but that's kind of start your of light. It's a little bit more complicated than that. There's some modulation involved. But that's kind of start your clock now. And then each of the axes in the – there's two orthogonal mirrors, spinning mirrors in the base station basically. And they sweep a beam of laser light across the world alternately. So it kind of sweeps, you know, in the X axis. We call them J and K, but in one direction and then orthogonally in the other direction.

And by timing from the flash to when you see the laser beam come past, you get an angle from the base station to where that sensor is in the world. And from those angles, you can then do a similar kind of problem, although we actually use very different math to the traditional computer vision solution. We use a very different tracking system. We can talk more about that later. But it's sort of the same problem. You've got a bunch of angles, you've got a bunch of projected angles from each base, and you can work out where you are in space from that. Because I have 32 detectors on my head, right? Correct, yes. there's 32 on the headset the controllers I think have 26 at the moment the 32 was chosen because it's a it's a nice round binary number and also it's about what you need for something that's kind of that shape we could get away with less but having some

redundancy is good if you know you put up your hand and block half of the constellation of sensors, then it's good to have some redundancy. Some redundancy. 32. That's a lot of angles. It's a lot of angles. I mean, I think the camera systems typically use 30 to 40, maybe more, dots. Many mocap systems use a lot less dots because they have a lot more cameras,

so they've got more redundancy kind of in the other way around, in view space. So you've got more ways of looking at the same dot. And they actually use slightly different math generally because they can do triangulation because normally they can see each dot from at least two cameras. So given this system, how well can you track something? That depends on how far away you are.

So like most tracking systems, it degrades with range and it sort of degrades with the position in two directions. So if you're looking out from the base station, range is the worst measurement. Much like station, range is the worst measurement. Much like GPS, height is the worst measurement. It's actually kind of a similar, although that's actually the mathematical dual of an angle system because it's a trilaterating system rather than a triangulating system, multilaterating.

Anyway. No, no, no, you're going to have to explain those words. Sorry. Okay. Then we'll go back. I don't know what a trilaterating system is. There's kind of two ways that you can, given measurements of something, you can either kind of measure how far away it is or you can measure what bearing angle it has to you.

So angular systems like lighthouse or cameras measure relative angles and use sort of like the old-fashioned resection problem that's used in surveying. They use big triangular grids of, you know, where you know all the angles and you know some of the distances and you can kind of solve for where all the vertices are in this triangulation net that's... It's a very old problem that's been known since, you know, surveying kind of just after the Egyptians kind of thing.

Then there's the other thing where you can measure distances. You only know distances. You have no idea about angle of arrival. So you can either do that by time difference of arrival or time of flight. And by doing that, you get a distance. GPS works this way. And in some ways, it scales better to larger things,

but also it means that you have to make measurements that are of significant resolution compared to the propagation speed of the radiation. For the speed of sound, that actually works pretty well. And there are many ultrasonic ranging systems out there that work in kind of room scale volumes because the speed of sound is pretty slow. But the speed of light is really fast.

And in a room scale volume, you're talking sub picosecond kind of resolution to get any kind of spatial resolution out of the system. In the case of Lighthouse, that could be done, but it would be pretty expensive. Let's put it that way. And for those kind of practical reasons, we measure angles instead of measuring distances. So that's the difference.

Trilaterating or multilaterating systems use distances, and triangulating or multi-angulating systems use angles. Okay. What were we talking about before that? I was asking how well it tracked. Oh, right. Right. Pico seconds.

What do pico seconds turn into? What do pico seconds turn into? Pico seconds, yeah, I mean, that's speed of light. What does that work out to? It's about like this Admiral Hopper used to say about the size of a grain of pepper, I think, is about the, is a pico second at the speed of light. That's about right. A nanosecond is about 11 inches. That's right. Yeah. So we measure in the nanosecond kind of region for Lighthouse because we only, the sweeping rates, you know, the thing is spinning at 60 hertz,

so it covers one tau or two pi in about 16.7 milliseconds. So if we have resolution down in the, you know, 40, 50 nanosecond, we can get angular resolutions around one part, 1.25 parts per million, somewhere around there is what we get. Wow. Then there's noise in the system. And noise in the system degrades that at about 5 to 25 microradians per meter. It works out anyway that at 5 meters,

you're probably getting about a quarter millimeter, one sigma in position. Which is on the face of it kind of incredible. It's pretty darn incredible. Yeah, it works pretty well. Distance, however, is computed by kind of the skinny triangle of the projection of all of those dots. So essentially the angular size of the constellation as seen from the base station gives you an idea of how far away it is. And that means that you've got this long, narrow triangle. So any kind of errors in the position that you've measured on those sensors get passed through an inverse tan and it ends up blowing up. So the error in range can be quite a lot larger than that. It can be 50 times larger potentially at a distance.

So you said it's advantageous to have two because if you turn your back, then certain sensors are shadowed and can't see one of the base stations. Is there any advantage to more than two? Oh, yes. As many as you can, really. But then that gets into some different kinds of scalability where where cameras and lighthouse have different advantages but first of all let's talk about multiple base stations i was only talking about one base station solution or you know one

projected angle same for camera okay in that case you've got you know poor range data but if you have two of these measurements particularly if they're near 90 degrees from each other, then the spatial, you know, the angular fix of one base station corrects the or compensates for the poor range of the other. So you get, in GPS terms, we call it geometrical dilution of precision for a multilaterating system. For an angulating system, it's kind of the same, but the math is kind of just inverted. And it means that if two things are at 90 degrees, you get pretty good, you know, fixes. And we certainly see that with lighthouse. When you can see both base stations, you get, you know, sub, you know, sub quarter millimeter, maybe down to a 10th of a millimeter fixes

within the volume. Do humans really need that level of precision? I mean, do you notice that? Yeah, you do notice it. Like if it was off by inches, particularly with your head, if it was off in the same way or if it was smooth, if the air is smooth over the volume, it's not such a big problem because you kind of, all of your senses are kind of relative as well except your eyes. And obviously your vestibular system is kind of like MEMS, IMU is kind of the same thing. It's all relative. So your brain normally kind of fixes, we believe, fixes its

position in space, you know, optically most of the time. So if the world's a little bit off, or there's a little bit of distortion in kind of the metric space that your tracking system provides, you probably won't notice it. And Lighthouse and all other tracking systems do have distortions in the volume that you can compensate for by calibration. We can talk more about that later. But if there was jumps or discontinuities, you know, big non-monotonicities. Really bad. Yeah, you'll notice that.

So, like, Lighthouse, like any system, will occasionally screw up and you will definitely notice. Yes, I've noticed. Oh, I'm on the Titanic. Everything's sinking. Yeah, the particularly bad ones is when you lose optical and, like, it starts having to rely on the IMU and it flies off, and that's not pleasant. But I've noticed that if you sit very, very still,

and I think there's some third-party program that measures the noise in the system and will tell you if you've got a good setup. And I think ours came back with sub-millimeter accuracy, which was something like that. But if you sit extremely still, you can kind of see a little bit of noise in your motion. So I think our eyes and our brains are very good at noticing that sort of thing.

And yet I was really impressed. There's the drawing game, the Google Tilt Brush, and you can draw or sculpt. It is sort of more accurate since you're in 3D. And you put the controller in one spot then you go back and you want to start a line from that same spot again and it's really close i mean it's just really repeatable and weird that way i was surprised i didn't expect that accuracy but now now i do part of that is obviously you're a human closing the loop as well so you're looking at the where the you know the point of of that is obviously you're a human closing the loop as well. So you're looking at where the point of your tool is. And if there was a little bit of error in the tracking system, like the constellation wasn't quite isotropic.

So when the system saw it from a different angle, it was off by like a quarter millimeter or something. You would just move a quarter millimeter to compensate for that. So humans have this kind of eye-hand coordination thing that obviously has been incredibly important in our evolution. And we're super good at compensating for those tiny little errors. And because you can't actually see your hands, obviously you're blind and you're only seeing the tool that you're wielding. It just seems to be a feature of our brains that we can really compensate for tiny errors quite effectively. So the lighthouse runs with, you said flash, and then it goes X lines and then Y lines, but these are all infrared. Have you ever replaced

it with a LED or laser you can see so you could see the pattern it's making? Oh yes. The early prototypes were done with red lasers, not red blinkers because that would probably give you a seizure, but the lasers have been read in the past and were in the early prototypes just so we could really debug it and understand it because there's no substitute for actually visualizing what's really going on. The sweep itself you can't really see. We have used high-speed cameras to look at the beam hitting the wall, for example. When you look at a base station that is invisible light

or if you look at it with an infrared camera, you can kind of see two dots on the rotors where the beam passes over your eyeball basically or passes over the camera lens. So as you move around, the position of those dots on the rotor in the base station kind of follows you, like those weird paintings with the eyes following you. It's kind of the same thing. So in our early base stations, you'd have these two red dots,

and they'd kind of just follow you as you moved around because you're essentially looking down the beam projection line all the way to the optics in the base station. Crazy. And these move, the lines move because you have motors in the system, because you're actually physically moving spinning mirrors. Yes. So mirrors are, well, obviously I'd love to be able to steer the beam some other way,

and we've certainly looked into that. Using a spinning mirror is actually the cheapest and most reliable that you can do right now. The physics also gives you a bunch of stuff for free because the rotor has inertia and by virtue of that inertia, the spin rate is extremely stable. So they're a little bit like pulsars. You have this little sweep. In fact, I was going to call it pulsar, but I ended up using pulsar for a different kind of tracking system that was as a time differencing rather than an angle differencing system. But anyway, as the thing spins around, the angle that we measure by timing comes purely from how many angles per second essentially that rotor is spinning at. So that spin rate has to be extremely

well controlled. And that's been one of the challenges in building the base station was to make a control system that could hold the rotor to parts per million and it's a mechanical system. And it does. And it does. Is that building on, I mean, laser scanners, that's a technology that has been around for a long time and I've worked with them in the past.

And they get more and more sophisticated. But it's kind of well-known. Did you build on that? Or was there, oh, my God, this isn't good enough. We have to do something else. Many of the laser scanners, I mean, obviously, they're normally quite expensive instruments. So we had to make something that was affordable.

There are many different kinds of angle measuring, there's theodolites, there's all kinds of systems out there that use spinning mirrors or spinning lasers. Many of them actually put the laser optics on the spinning bus of whatever's actually scanning. And doing that is obviously better because you can align everything fairly precisely, you can potentially actively align everything in the factory, but that's prohibitively expensive for our application. So what we had to do, and I guess kind of one of the main inventions that I came up with for Lighthouse was to use a diverting mirror,

a spinning diverting mirror and a line generating optic that were fairly cheap, off the shelf kind of things, and come up with ways of compensating for the non-idealities of the alignment of that system. So to build something at a consumer price, we had to do the math and do all the investigations into taking something that was very much a non-ideal system that was sort of built with some tolerances and turn into something that could give us very good data. And how do you manufacture to get that sort of, I mean, it must be a pretty fine-tuned calibration. How do you even, how do you even? And it's pretty good. You know, you probably get accuracy to about three or four inches absolute without calibration because, you know, it's assembled reasonably well.

And HTC does a particularly good job. They can actually build base stations better than we can, our early prototypes. Many of the prototypes, of course, they're assembled by myself or someone else. They were, you know, just sort of thrown together and would kind of glue the lenses in at roughly the right angle and the mirrors would be roughly the right place.

But the motors that we now use are very, very good motors. They're the same kind of motors that you have in hard drives. So they've got, you know, tolerances in the microns in terms of their run out and their axial play and things like that. And on top of that, we have, you know, mostly polymer rather than, and there's some metals involved so that there's some tolerances in the mechanical assembly of all of that. And that's kind of what we have to calibrate out. So calibration, we actually use similar to how you would calibrate other optical systems. It's essentially nonlinear regression, right? You collect a bunch of data over the volume, and you have some kind of mathematical model that says, this is how we believe the base station should work. And there's a bunch of parameters that say this parameter compensates for one non-ideality or the other.

And then you solve for the values of those coefficients, basically, those calibration coefficients, which minimize the error over the whole set of data that you've collected. And that's how you would calibrate most systems of any real type in the world. Newton's non-linear least squares regression kind of stuff is kind of universal in that respect. But the model itself is pretty straightforward. We have, for example, one of the parameters is the exact angle when that flash happens. That when that flash that says, okay, I'm at angle zero, basically,

there's a mark on the rotor, there's an optical pickup that determines when that flash happens. And the positioning of that, that is actually just a little piece of shiny aluminium tape that's put on the rotor. And the positioning of that, you know, is not accurate to within sub-millimetre kind of thing, kind of positioning just from manufacture. So we have to calibrate that out. And it's one of the parameters

we actually calibrate out really, really well. Yes, because once you know, when you have, you can talk to the motor subsystem and say, okay, tell me when you're going to hit that zero angle and you can have a camera that waits for that flash, and then you just take the time difference of those two, and now you're calibrated on that particular parameter.

Yeah, and that is kind of how we did it initially, but it's not how we do it now. We actually don't just take a singular measurement like that because alignment of the base station in the camera or the sensor would be very, very difficult to get them because the system, you know, it can do five microradian kind of resolution. Lining up something to five microradian resolution would require like a theodolite and would need reference fiducials and it would be almost impossible to do that in a production line.

So we take a bunch of data over the whole volume and we know, we do that in a production line. So we take a bunch of data over the whole volume and we do that with a normal tracked object, just like a headset. We sort of wave a headset around in the volume in front of the base station. And by collecting all of that data, we know that the headset didn't change shape significantly, although you can actually tell that it changes shape from the fact that you're moving it around and gravitational loading. But anyway, you get a whole bunch of that data and you look for, you solve for the number that says that makes sense, basically. So if those angles are wrong, the world is slightly distorted,

the metric space is distorted, so you solve for the numbers that flatten that metric space. When you were prototyping this, was it a bunch of free space stuff on optical benches and parts from Thor Labs, or did you go straight to kind of, you know, a form factor small device from off the shelf parts? The very first version was actually a 3D printed plastic thing.

Really? Yeah. You can actually make a very crude lighthouse. Anyone could make a crude lighthouse for a little robot that ran around on the floor. I've actually been thinking about publishing exactly how to do that because it's a simple two-dimensional problem and it would actually be super educational.

I could imagine a bunch of kids building one for fun. But the first version was just that. It was like a laser pointer that I took apart that I got off eBay and I took a vibration I got off eBay, and I took a vibration motor out of an Xbox controller, and I used that to spin a little mirror cell, and it was all 3D-printed plastic. And I assembled it, and I built a receiver,

and it gave me like 0.1 degree resolution because of all the noises in the system. It wasn't closed-looped. It was just spinning as fast as it span and it had cable sync. It didn't have optical sync at that point. But it was kind of the proof of concept. Then the next version after that was we sawed up some hard drives.

We literally took some hard drives out of the dead hard drive bin and we took the platters out and we sawed the front off it and stuck some... At that point, we went to decent mirrors that we got from thor i think um but we still used like ebay line generator optics and a bunch of um one of the one of the

first rf receivers because there's some rf modulation on the uh laser beam was uh was one of those am radio kits that that ben krasnow apart and, you know, he kind of plugged a photodiode on one end of that and went through and changed a bunch of stuff and used that to build the receiver. And then from that, we started to build more serious ones that were actually like machined aluminium and machined brass for the rotors. And a couple of versions beyond that, we were at the point where, okay, we need to make this cheap and simple.

So we, you know, we had castings and post machine. We tried to make it as cheap and simple as possible. And really the motors and the electronics and the lasers are probably the most expensive part of it. That's just amazing. Okay. So if you have all of this really precise location ability, what do you need an inertial system for? Well, it only spins at 60 hertz, right? So although you get two flashes out of that, so you get 120 half updates per second per base station, and you're rendering it like 90 hertz, and you need something essentially to fill in the gaps

in between the optical systems. So the IMU, we can sample at a thousand hertz, right? We can get data out of it at a kilohertz. And we use that as kind of the primary high frequency information in the system. If you think about it in terms of like frequency, then all of the really fine high frequency short interval updates come from the inertial system and the low frequency close to dc information about where you actually physically are in space that comes from the optical system and they get fused together in a kalman filter so it must be i mean kilohertz updates that's more gyros than accelerometers at that point i feel like I should ring a bell anytime somebody says Kalman filter. Yeah, unfortunately

the chap died recently. I wonder if he appreciates just how important his contribution has been to people, all of our tech, really. Yeah, I mean it's been pretty amazing that he made this algorithm that was, it's not that complicated but we do so much with it. We make it so complicated with all of the ways we use it. Yeah. It's so general.

If anything that's roughly normal, it's super applicable and it's just so elegant. Yeah. Okay. Okay, so you put in gyro at 1 kHz, you put in visual light area at 120 Hz, because there's two lighthouses at 60 Hz each. And then you put in the accelerometer data whenever it's available, probably closer to 500 Hz. Yeah, okay, you can go pretty fast with that so when i'm when i'm whipping my head around it does track yep it's it's actually pretty good at estimating um and there's been some people like doc okay i think did a a rather nice youtube videos a set of YouTube videos on Lighthouse where he collected the data, the raw data straight out of kind of the tracking API.

He kind of bypassed a bunch of stuff in SteamVR and he could show you the updates where it would correct in one direction and it would correct in the other. And you could see the IMU had drifted a little bit and then the optical would push it in the right direction and then it would push it in the opposite orthogonal direction. And he also showed you the data, the kind of the noise cloud when you had only a single base station versus two base stations. And you can kind of see a lot about

how the system works from that as treating as a black box. We will get a link to that in the show notes. So does this Kalman filter run on the Cortex-M0s in the headset, or is it on my computer, or is there an FPGA in there that does it? It's in the cloud. It's on the host at the moment. So the Kalman filter does, you know, there's a fair bit of math involved. There's an inversion of some pretty big matrices. It could and probably will eventually be baked down to run on the tracked object. It's particularly advantageous for things like the controllers that have a limited bandwidth connection.

They currently send updates at about 250 hertz at the moment, whereas the headset run as a kilohertz for the IMU. And a lot of that is just purely because you need to ship that data over radio, and to do that, a kilohertz is a lot. That said, tracking on the head has to be way better than tracking on the hands because, again, eyes are more sensitive than that positioning. It has to be more sensitive because vomiting. Exactly.

If your hand flies off, it's annoying, but if your head flies off, yeah, it's not pleasant um but you do have some m0s on the headset and on the controllers i think although i'm not sure about that yes uh what kind of math are you doing there is it just coordinating everything and communicating yeah primarily that that's just that that's the watchman, basically. One of those M0s in the headset and in the controllers is primarily reading the IMU, getting the data out of the FPGA from the optical subsystem. So yeah, you're right. There is an FPGA in there. The FPGA is kind of doing the high-speed timing of the signals coming from the sensors. So each sensor outputs

like an LVTTL signal that comes back to an FPGA. That FPGA has a high-speed clock that is the time base basically for all of the optical system. The IMU also happens to feed into that FPGA. So we get the IMU data in the same time base because time is super important in any kind of tracking system. So everything comes back to the host over USB or over radio and is in the individual time base of each one of those Watchman receivers. And then the host side code uses that information to actually solve for the position of those objects. And I know that you use three ST Cortex-M0s and one NXP Cortex-M0. I have to ask you this.

Do you hate your software engineers? Yeah, the NXP selection actually has some history and some technical reasons why we went with NXP for the Watchmen. The Steam controller is based on on nxp 11u37 is exactly the same although we started off with a slightly different chip but we ended up in the same kind of region the nice thing about the the nxp lpc 11u you know xx devices is that they have a usb bootloader so they're essentially unbrickable it's in the rom and if you you know bridge and bridge and bring a pin down when the thing powers up, it will come up on USBs and mass storage device. Many other chips have this feature, but that was one nice feature that the Steam controller team selected that thing for

because they could wire that up to one of the buttons on the controller, and no matter what the person did to their thing or what terrible software update we might have accidentally shipped, we could always unbrick it. So that was one reason why we chose the NXP. The bad thing was LPC open wasn't a thing at that point. And a lot of the example code that we got from NXP was really, really terrible. So when I first came here, I was looking at this stuff going, wow, was this like test vectors?

Was this like some guy just writing some code to test the peripherals? It was terrible. There was like hard-coded clock rates in some of the timing routines and the USB stack is an abomination. But anyway. It's not just me then. No, it's not just you. The NXPs, I actually don't know about LPC Open now. I know that they put a lot of

effort into it. I actually had one of their reps was here and the poor guy, look on his face when I told him what I thought of it was actually priceless. But I think they've invested quite a lot of effort in making LPC Open better. That said, I haven't used it, but I know that the controller team has and I think they've actually migrated to it. But one of the first things I did when I wrote the firmware for the base station and the original firmware for the Watchmen, so a lot of the time, a couple of weeks, I spent writing drivers for every single peripheral on that chip. So I know the 11UXX pretty well because I essentially wrote a driver for every piece of hardware it has.

But then you went to the ST because it's a little better supported? No, the ST that's in the headset, I think there's an ST in the headset. Yeah, I think that was HTC's selection. So the display subsystem in the headset is based on our reference design, but they wanted to do some stuff themselves. So because it's always complicated when you have two parties trying to collaborate over a code base, they essentially picked their own microcontroller

and did their own subsystem. So their code kind of runs in that stuff and our code runs in our micro. And that's just why we ended up with two micros. That'll probably get cost reduced away at some point. So you mentioned USB and that takes data over to the computer. But USB, it's not real time.

Does that cause problems or do you just have enough control and have specced? Isn't there some sort of isochronous mode or something? I've gone way back into the depths of my memory now. We're actually using, yeah, it does cause some problems, particularly because there are many USB implementations out there, particularly USB 3.0 XHCI drivers that are just terrible, terrible, terrible implementations. And some of them just have broken silicon as well,

not just the drivers in Windows. But anyway, in terms of the, like the latency is a little bit unpredictable, but it's generally, you know, a few milliseconds and we can kind of deal with that. It does have, it's kind of one of the parts of the system that we don't have great control over because the PC platform, there's so many different variations of PC platform and different chipsets and things out there. In terms of time, everything sort of runs in Watchman time or something derived from Watchman time that's kind of a host time plus some boxable latency really on the host.

We don't use isochronous. We're actually using HID, believe it or not. Yeah, but HID is interrupt-driven, and we can get one kilohertz data out of it, although Windows, at least previously, I think Windows 7 and below had a bug that if you had like a one kilohertz HID device, it would just refuse to boot if that device was plugged in.

It's too fast. Yeah, it's a super common problem. Like a bunch of mice in the world break this as well. And it's something strange to do with power management on the USB bus as well. We eventually worked around it, but it's a pretty common problem. USB is one of those things that you think, wow, you know, it's all nailed down. It's a pretty simple, straightforward standard.

It's universal. Yeah. Everybody's making their own standard there. Yeah. Yeah, USB has been, it's probably one of the most, you know, challenging parts of the system in terms of just support because there are so many broken implementations out there.

Yeah. And so USB takes over your sensor data from your headset and then HDMI then sends the display data back over a different cable. Do you send data over USB as well from the computer? sends the display data back over a different cable. Do you send data over USB as well from the computer? Yeah, there's a little bit of data sent over. There's actually a way to retrieve the calibration information. So, for example, the calibration information for the optical and the tracking subsystem is stored in the headset.

We can retrieve that via USB. There's some other sensor infamously auxiliary sensors like the proximity sensor and the IPD measurement sensor. There's also some kind of power and optical display control stuff as well that goes over USB. And of course, there's the ability to upgrade everything, right? Firmware upgrade. So there is also an unused USB slot in the headset,

which of course, you said that one of the science fiction concepts that might become real in our lifetime is the mind. I don't want my brain going over USB. But I mean, plug in somebody else's brain into my headset. It'll work out fine, I'm sure. What is that USB slot for? That's an auxiliary USB for whatever you want to put in there. So one of the things we kind of do here at Valve is we try and build our stuff as hackable as possible.

We try and make it very hacker-friendly and flexible and expandable. Even the mechanical design of the headset, we asked HTC when they were coming up with it to make it the strap removable so that you could potentially make your own strap. The USB is there purely for whatever peripherals you might want to add to it.

You can put different kinds of tracking systems. I know some people put the leap motion, the camera system that they have on the front of the headset. So that's a general purpose USB 2.0 port for whatever you want to try and hang off the thing. And we encourage people to play with it. I've seen people put those USB powered fans. That'd be kind of nice sometimes.

Yeah, I love that. Some of them are just like a bear spinning fan blade, which seems like it could be bad too. So this is actually hubbed to the computer. It doesn't modify the local headset at all. It's just a USB to the computer. Correct. There's a seven-port hub in there. Okay. That many of the things hang off.

And one of the spare port was basically bought out for auxiliary use. I'm thinking about all of the things I have wanted added, more trackable objects. I really sometimes wish I knew where my feet were. But I don't think I want to wire them to USB. I want to know where my drink is. Yes. Can we have tags that will let us go on to drink and we can find it in the world? There's a couple of ways of doing that.

One of which might actually be to use the camera. It has a camera on the headset, and the camera is very well calibrated. We know exactly where it is, and it's tracked by the tracking system that's in the headset. So one, you know, lighthouse receivers are fine. We're obviously going to have more lighthouse receivers in the world very soon.

But for simple things like finding your drink, maybe you want to put a fiducial on the drink. You just stick a sticker on your mug or have a mug that has a fiddy on it and we use the camera to work out where it is relative and we put it into the virtual world. There's obviously some software challenges. Whenever you had the same thing with extra trackers

like for your feet, the Vive was given to developers essentially as a complete sort of system as this is the minimum spec. Everyone will have hand input. Everyone will have track displays. And you can go and build your games based on that or build your experiences based on that. But when you start adding all these third-party peripherals,

then many of those in terms of the market for a third- peripheral, will live or die based on whether people support it. So one of the challenges for us and the challenges for all the developers out there is when we get this community of different devices that people are going to make for the system is how to expose them as an API so that people can actually use them and how developers will work them into whatever kind of experience that they're going to do. And that's a challenge that we're going to have a crack at solving very soon. So you mentioned that you started out with the goal to have things be somewhat hackable,

and you also talked about a simple example kind of thing for kids, two-dimensional robot control. So I guess I'm curious how you would recommend somebody get into developing a tracked device, whether for hacking and do-it-yourself projects or as a product? Is there a pathway to do that that's, you know, okay, you need these parts and you need this kind of software or is it kind of on your own figure it out kid so right now you're kind of on your own and what i've actually you know fertilized the world a little bit i've been i posted on twitter the circuit diagram for a census um you know circuit there's been many people who've reversed engineered pieces of it and there's some YouTube videos out there and there's a website that this guy's made where he went and decompiled the firmware. I didn't blow the bits on any of the firmware. So you can suck the firmware off the things and reverse engineer it. That was deliberate. We want all of our devices to be super hackable.

So all of the code is out there essentially in the public domain, in binary anyway, not in source obviously, but not now. And people have gone through and reverse engineered. I actually had some people send me some bug fixes for the base station that they found by reverse engineering. So there's been a community of people out there that have been interested enough in the technology that they've already gone through and learnt enough that I'm actually surprised no one's built their own tracker with it yet. Some people have come pretty close. They've

got to the point where they understand the emissions of the base station well enough that they probably could build a tracker. They just haven't gone and done the solution for tracking. Now, in the near future, there is going to be a reference design and a development platform that will be available, and there will be support for people that want to do this commercially. Chris and I are stunned to say that because we're now going to stop playing with Unity and trying to design games and start trying to play with what we can do with your hardware. We'll get as far as we did with that as we did with the other part. Well, stay tuned. There's going to be an announcement very soon about exactly how you might go about doing that.

So it sounded like I could already mark my drink by putting a sticker on it and using the camera, but then I still have to write a game so that I can get to my drink. You'd have to write software that recognized the sticker. But I couldn't put that, like when I pause the game, I can't just find my fiducial. No, the game would have to support it. We could probably put it in the, that gray room that you go into,

we call it the compositor, although there's been some, there's been a competition to see if you can come up with a better name for that. But that room kind of, we try to keep it as simple as possible. There's a whole bunch of things we would love to put in there, but it always has to run at frame rate no matter what, even with people that have poor machines. And we're a little bit hesitant to overload it with too much functionality. But I think something like if we do implement, and I know there are people working on this, CV, computer vision style add-ons for SteamVR, then it's quite possible that we would add a generic, go print this QR code or this FIDI and go and stick it on whatever you want and have that passed through. Or at least punch a hole through because you know how you can go into the Tron mode kind of thing with the camera? Right.

We could potentially just have some kind of recognizable object that just punches a hole through and that would be fairly lightweight and something that we could put in the composite and could run all the time. It's like inverse augmented reality. Yeah. It's like a mediated, some mix in between AR and VR. Something real bleeding through the virtual. Yeah. Yeah, it's like a mediated, some mix in between AR and VR.

Something real bleeding through the virtual. Yeah, so I'm not quite sure how practical that would be. It's something I'd love to see. If we don't implement it, I'm sure someone in the community will implement it. I know that someone here is working on publishing how to access the camera for computer vision applications because the Vive actually makes a very good pseudo AR or CV dev platform because you have a track camera. That's amazing. I'm still sort of boggled by all the possibilities here. Do you actually get

to play it very often? I've done a little tiny bit of Unity and now you're opening worlds but for the most part i have been a consumer but of course this was a work um purchase a business purchase i'll cut that um so i do just play but i wonder thinking of products i've developed i haven't really gotten to play with them because when I play with them all I ever see are bugs do you ever get to play with VR uh I do get to play with VR not as much as I would like sometimes uh because I'm busy like making VR but uh yeah I have a Vive at home um I've had a couple of different generations of the Vive at

home. I'm actually still using very old generation base stations. I've never swapped out my bases. I have the early prototype of the optical interbase sink, like hanging circuit board, hanging off one of my other base stations, just sort of soldered into the back of it. And it's kind of like the, you know, the cobbler always has the worst shoes kind of thing. It's definitely that. Like, and there's that tendency that you know exactly what to look for. So you see all the little problems. That's true. But you can just relax and enjoy it.

Like I like space pirate training. You mentioned it before. I like Zen Blade. There's a lot of things that I'll, you know, I'll just put on sometimes when I come home at some ungodly hour and just play for a few minutes before I go to bed but I'm definitely one on the team here of many people that doesn't play it that much. I'm more often in the lab trying to build something than playing too much with it. The software people, obviously they have to play with it all the time to test it. So in many ways they have a better job. I tend to use it for exercise. I didn't

expect that. I didn't expect to really play it at all. Chris got it and then he had it for like an hour before I was like, okay, we're moving this to the living room. But I've been talking to people about exercise lately and how Fitbits made me want to complete the next milestone. It would get me to walk an extra 10% because it would then throw me a little party on my wrist. And then Pokemon Go, which I have to admit is a little fun. It gets me moving. It gets me started. And that's really great.

You know, sometimes you're just, it's the end of the workday and you're tired. You don't want to do anything. And fine, I will go capture a stupid little creature. And I like that. And the VR system has some of that, but it's also got a, when I am in the system, I am 100% there. I'm not thinking about anything, but whatever is in this game. It's more of a Zen thing than just about anything else I do. Have you heard that from other people or is this just me? No, it's not just you. There are a lot of people that enjoy it as a recreation and exercise tool. Certainly what I read on the forums, you are not alone.

It's funny what you say about other things like the AR games that are coming out like Pokemon Go and that kind of mixed or mediated reality so you can have either the complete blinder experience where everything is absolutely virtual like with the Vive or you have things where you bring artificial elements into the real world. And I think that as the technology improves, we'll just see more and more of that, right? You'll be able to dial it from completely artificial to completely real and anything

in between. It's amazing that it used to be your parents would tell you to go out and play and not sit in front of your computer and play games. And now playing games may be how we exercise. Yeah, absolutely. It's strange, isn't it? Because you've always got like the couch potato kind of thing, a gamer, he just sits there, never gets a tan or anything. But that may be, as a stereotype, may have to change very soon. It already is changing, I think. So I don't want to keep you too long because you have, you know, fiducials to start printing so I can find my drink and lots of other neat things coming up. Chris, do you have any last questions?

I did want to ask, when you started all this, did you already have a background in kind of tracking stuff or did you learn all this on the fly? I mean, you talked about the triangulation and trilateration system. Did you have familiarity with GPS already? Not particularly. I mean, I did have, you know, I've been an engineer for many years and I've been obviously curious about everything, kind of read Wikipedia religiously and learn everything I can. But no, I didn't really have a background in,

you know, in state estimation or any kind of thing that would have, or navigation or, no, is the simple answer. There you go, kids. Read a book. Yeah. Working at Valve is very much like that, right? On a daily basis, you will do very little of what is your core competency. Yeah. So that's why when we try and hire people, we try and just pick smart people who know their stuff.

We call it the T-shaped person that has some technical depth that is their own, you know, whatever they're really good at, know it all the way down to the, you know, the atoms or whatever. And then just general breadth because you're making new stuff, you're making up things that don't exist in the world, right? There is no course that says, you know, Lighthouse 101, although there probably will be in the future. You just have to go from first principles and any reasonable engineer should really be able to do that. They should have their thing that they're really good at

and they should have enough competence in everything to be able to invent the future. Yes, I agree. Oh, and it looks like I do have one more question. Gravitational loading? You can see gravitational loading on these things? Really? Oh, yes. Yeah. There's a long story of the many discoveries about how stiff matter that looks

solid is not that we find with lighthouse. So, yeah, if you put a lighthouse on the corner of the table and you lean on the table, the table deflects enough that you can measure it. One of our calibration things, we have a robot move a tracked object around in front of the base station to take a bunch of measurements, and we had a little rubber sorbethane pad that we put the base station on and would have the robot execute a move, and it was doing a circle, right? And it didn't quite meet up at the top when it returned back to its starting place. And for a long time, we're going, why does it not meet up?

And it turned out that the base station was sinking ever so slightly, you know, 20 microns or something into the rubber during the course of this test. So we can resolve all kinds of tiny, tiny motions because Lighthouse, you know, it's noisy but it's zero mean. So if you keep integrating it forever basically the longer you integrate the better it gets like gps yeah and so how often did you attribute these things like that to errors in the software errors in the hardware before you realized, no, no, it's sort of an

error in the world. It happens a lot. There are many times where we spend weeks chasing down something that turns out to be some unexpected thing, some unexpected feature. There's been some interesting, like optics is a crazy difficult discipline. There are so many compromises and so many things you didn't even think of. We've chased down stray light in lenses that we didn't even, you know, imagine this path could exist but, you know, this path existed. The universe is one of those things that it will remind you that, you know, it's in charge.

Your models and everything else in your perceptions about how it should work are often dead wrong. So we're very much about making empirical measurements and studying these systems. And we still have many, many things about – this is all brand new, right? It hasn't been around in the world very long. So as a first-generation thing, we keep finding interesting features and interesting non-idealities that we have to learn how to correct every day. And they said physics and calculus wouldn't be important in our careers. That's when you asked me that question much earlier about whether

science, engineering, maths, what was the other one? Technology. Technology, right. All of the above, right? Like you've got to be, science kind of gets you the information that you can do, engineering and everything's talked about in terms of math. So you need all of them. That's what I meant by all of the above. All right. I'll accept that answer now. Do you have any last thoughts you'd like to leave us with well i yeah you said this would be a hard question and you weren't wrong um i i think the future is super exciting for all this stuff this is kind of you know generation 0.9 really you know it was the minimum thing that could possibly be made for the consumer and i'm just super excited to see where it goes.

Every day when I see new experiences or new games that people have created, the creativity of humans and what you can do when you give them a new media is just mind-blowing, and I'm super excited. Me too. Me too, very much. Well, my guest has been Alan Yates, hardware engineer at Valve. Thank you so much for being here. Thank you very much. It's been a lot of fun.

Thank you also to iFixit for their very informative teardown. You didn't think I just knew all this stuff, did you? Also, thank you to Andre Cicak for giving me some great questions. And thank you to Christopher for producing and co-hosting. Finally, of course, thank you for listening. Hit the contact link or email show at embedded if you'd like to say hello. As usual, I do have a final thought to leave you with. This one from R. Buckminster Fuller.

You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete. Embedded FM is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting Thank you.