Embedded - 23: Go For Everything I Want

Episode Date: October 16, 2013

 Jeri Ellsworth joins Elecia to talk about about co-founding Technical Illusions and their virtual and augmented reality product CastAR. Jeri gives an in-depth introduction to virtual reality, augmented reality and motion sickness. They also talk about hardware engineers working with software engineers, the CastAR's Kickstarter, children's toys, and tagging sharks for science.

Welcome to Making Embedded Systems, the show for people who love gadgets. I'm your host, Alacia White, and I'm thrilled with my guest today, Jerry Ellsworth, talking about reality, both augmented and virtual. Hi, Jerry. Thank you for joining me. Oh, thanks for having me on. You've been called an electronics hacker, a chip maker, a race car builder, a pinball machine enthusiast, a blowing up stuffer, blowing stuff upper. How do you describe yourself? Well, okay, describing myself, I'd say that I go for everything that I want. I don't let anything scare me. So for instance, the race cars, I knew nothing about race cars when I wanted to learn how to build a race car.

And I just went out, found the right people to mentor me, and I figured out how to do it. And, you know, likewise with all the chip design and various other things I've done, I just went out and found the right people that would mentor me and figured it out on my own. That is quite a skill. And you've been a tinkerer since childhood, right? Oh, absolutely. My father used to be so, would get so upset with me because I would take all my toys apart. As soon as I got old enough to where I could use a screwdriver, I would take my toys and I would start prying them apart. Mostly I think it was because I didn't like the idea that there was anything out in the world that I didn't understand.

I didn't like the concept of like magical things. So I wanted to understand like, you know, even down to pulling the legs off of grasshoppers. I kind of wanted to know how that stuff worked. I bet the grasshoppers didn't give up their secrets as well as some other things did. No, yeah. Tried to add a couple legs, you know, transplants. That didn't work either. So I learned some valuable lessons about biology.

Well, and maybe that will be applied to robotics in the future. Yeah, perhaps. But right now it's augmented reality. The Kickstarter went up yesterday for CastAR. Is that how you say it? Correct, CastAR. And AR would be for augmented reality, but it does a lot more than that. It does.

And actually that was kind of, I guess I can go into that a little later, but that was a sticking point for us. What do we call it? Because it does more than augmented reality, even though it originally was conceived as a way to do augmented reality. And the backstory on it is about two years ago, I was hired to a company called Valve Software. They're a very big software company. And they were worried that their stronghold in the PC gaming space was being eroded by tablets and consoles and things and people moving away from PCs.

So they wanted to develop their own hardware platform. And so I was their first hardware hire. And I started their hardware department. And so I came in and we started recruiting all the best talent we could find. But that's tough being the only hardware person because you deal with software people who don't understand that hardware is physical and it takes time to get. It's not just people time, it's shipping time. To get the electrons from here to there, you have to also

have the protons and the neutrons and you have to actually ship things, and software engineers don't always get that. Holes and electrons, right? You've hit on something huge. Our team started to form. We started having some issues as far as folks that were in control in the company not understanding what it takes to do hardware. So there was always this divide between the software folks and the hardware folks.

In software, they were very accustomed to iterating very quickly, so they could throw stuff out with lots of bugs and then fix the bugs as they come up. It's a little harder to do, and fixing bugs means soldering something, let alone new chips. Exactly, or cutting new tools for plastic or spinning a new board.

And so there was like this tension. But we were working through it all, and we made a bunch of really cool input-output devices. We explored every aspect of how do you interact with games. So we were doing things like measuring people's galvanic skin response to see what their emotional state was. It was kind of neat where if someone got scared or excited in some way, their skin resistance changes, and you can detect that. And so we were trying to evaluate if that was useful in gaming.

If you send a zombie towards someone and you see that their skin resistance goes down, you know, maybe you send another zombie and you can heighten the game experience. I wish we had horror movies like this because my tolerance is so low. I could still say, well, I've seen the horror movie, but it would all be like pink bunnies jumping around. Yeah, exactly. So that was an interesting challenge as we were going down all these different paths at galvanic skin response,

looking at people's eye position to see what they're looking at, various form factors for game controllers, for input and different methods for input. This is very scientific and researchy. It was. It was very much a research lab, but many of us in the company, in the hardware group, were accustomed to shipping things.

Oh, I understand that. Yes. I love to ship things. Yeah. So there was also a bit of a divide between some of the folks that looked at it as research, and then some people that looked at it as like we're going to ship products. And that's actually where kind of the rub with the augmented reality stuff that we're working on now outside of Valve software is that I took an approach that we're going to do this in a process that we're going to ship a product. So we're going to put a stake in the ground at some point. We're going to say good is good enough. And these are the feature

sets. And now we're going to go into the design for manufacturing stage where we iterate on improving reliability and manufacturability and ergonomics and stuff like that and quit trying to find the silver bullet for what's any more fun. Quit trying to make it perfect and trying to make it consumer ready. Exactly. It's a different set of criteria. Yeah, so the difference between research and shipping a product is the point where you lock the hardware feature set and then it becomes like a six-month design cycle

of iterating on making tools and stuff like that. Wow. And then you, you left Valve because the pressure fired me. All right. Yeah. To be completely honest, you know, this, this tension between folks in the company and, um, and us shipping and when to ship and what was research and what wasn't research put me at odds against some of the folks. And so one day I was very surprised. I came in and I met someone as I was going up the elevator. And they said, can you believe what they did to so-and-so? They fired him today.

And I'm like, what? And, you know, I kind of felt like mother hen for the whole group because I was the one that started the whole thing. So I went upstairs and I was going to like, you know. Better rehire him or I quit. Yeah, yeah. I was, you know, ultimatums is what I was going to do. I didn't, I was really upset. I come storming in and then I, my colleague, Rick Johnson,

who is now working with me on Cast AR, my co-founder, he, as soon as I walked in the door, he's like, I got fired today. And I'm like, what? What's going on? He goes, you're fired too. They just haven't told you yet. And I can't believe it. And like all this emotion, because we developed this augmented reality set of glasses. And it was really the play testing, people really liked it. And you got it to work. It wasn't like it was really the play testing people really liked it the cost got it to work it wasn't like it was some theory that was might someday potentially maybe in the future work it was actually hold it and here's our path to manufacturing exactly it was it was play testing well people liked it they were coming into the hardware lab and they were they wouldn't

leave they would stop, start playing these multiplayer games with these AR glasses and they would be very disruptive actually because it was so entertaining. And so I was devastated. I mean, I was really upset that this was happening. And so I had like two or three hours to sit there and stew on it, knowing that I was going to get fired. And, you know, I worked out in my head, like what I'm going to sit there and stew on it knowing that I was going to get fired and you know I worked out in my head like what I'm going to do what you know nasty things I was going to say to the yeah I mean I totally understand one time I was writing my resignation letter and they laid me off and I'm like I'm so glad I hadn't pushed print yet yeah now you get the uh

severance package but the the yes the stewing wow yeah so you know i had all this stuff worked out in my head i'm like i'm gonna go tell gabe who was the founder of the company like i can't believe you did this to me and you you're destroying this project and um it was quite different when i actually got there because when i walked in the door it's like half crying and like sniffling. And then I was kind of vacillating between like my aggressive and then my sad. And so I pretty much said, Gabe, you know, this project's dead.

It's dead. You fired everyone on the project and you should just give it to us and let us develop outside the company or you should fund it outside the company because it's a good idea. A spin out. Yeah.

So that was what I approached him with. And he just turned to the lawyer who was in the room and said, let's just set up a technology transfer and it's yours. And so I felt a little bit better at that point. I got less weepy at the point of him saying that we could have it. Because at least something I'd poured my soul into for eight months, um, was not going to completely die. So I went back down and told Rick about it. And we went over to my house that night and we drank an entire bottle of wine and we watched Dukes of Hazzard of all things, just like come try to comfort ourselves as much as possible. But we planned like, we're going to do this come hell

or high water you know we're going to figure out how to get this to market and so the next day I took all of my tools and oscilloscopes and laser cutter and and milling machine and stuff over to Rick's house and we set up in his living room and that's where we've been developing it this entire time there's it looks like a bomb went off in his living room. Ah, the engineering grenade. Yes, I've seen that go off in my garage a few times, yes. Well, that's kind of heartbreaking. But they did, I mean, they experimented with hardware and that didn't work for them.

Well, they're continuing to do hardware, but... But not yours, because yours is much too cool for them. Yeah, that's the way I like to think about it. And so we've been talking about it, and we mentioned CastAR, but tell me about it. What does it do? You have these glasses here, and... Okay, I'll describe how I discovered the technique that we're using.

It was a complete accident. So there's different types of augmented reality glasses. There's like Google Glass, which is a near-to-eye display. And that's kind of a heads-up display. It shows you things that are nearby, and you can take a picture, whatever. Yeah, it doesn't... For instance, Google Glass is a near-to-eye display.

It has a camera. It has a thing called a waveguide, which transfers the image out to right in front of your eye. And you have to focus through this optic system to a display which is near to your eye. And with those systems, they're often stressful for the user because you have to set a focal depth and, you know, the user and it's fixed. So the user has to kind of get accustomed to focusing at that depth, no matter what they're looking at. For instance, if you had a stereo version, like there's an, uh, another near to eye display called Oculus Rift, which. Oh yeah. There's one right over there. Oh, cool. You have one. They're pretty neat.

Although I did want to ask about them because they warn you that motion sickness, but your Kickstarter doesn't. It said no motion sickness, so I'm like totally excited about that. I get sick walking, so this should be awesome. Oh, but I'm leaning up to that. That's actually a big part of it. So we were researching virtual reality at Valve

with these totally immersive displays where you have a display per eye, and it presents you with a stereoscopic 3D image. So in virtual reality, your eyes are completely blocked. You don't see the world around you, but you're getting presented these stereoscopic images of an entire synthesized world. Now, part of the reason people have eye strain, get headaches and get nauseous, there's a lot of complicated things going on. This is what we were researching at Valve. For instance, if you set your focal depth,

so you have this little micro display in front of your eyes, and you set that at infinity, that's what they call it. It's like three or four meters out. That's easy for the user to gaze out, just as if they're gazing out into, you know, way out into the distance and they can see the image. But the problem is when you start presenting a stereo image in the vergence where it seems like an object is coming very close to your nose or to your face, your brain has a muscle memory in it to when your eyes converge inward, your brain remembers where it should be focusing, and now there's a conflict

because your focus of the display is at infinity, but actually your muscle memory is trying to focus at a foot away. It's like if I had an automatic camera and told it it should maintain focus on this image but you're not allowed to actually change the focus. Yeah. Even though the camera may note that it's coming towards it and it should go into macro mode instead I've said no you can't and I could see as my brain would go dear camera you have fouled up. There is no way to fix this. So that causes headaches.

Oh, yeah. That's horrible. And then motion sickness is another complicated thing. You have your inner ear, which gives you a sense of your balance and how your head is positioned, whether it's level or not. And the proprioception of where your body is. And if you don't have a really accurate tracking system on your head mounted display, and if you're totally cut off from the world, and you start presenting images to your brain that don't match your inner ear, for instance, if you move your head one millimeter, but actually the computer generates, you know, 20 millimeter shift in the image, it's like being on a boat you know the boat

moves the horizon's not moving as much as the boat's moving you know there's these weird things where your inner ear conflicts with what you're seeing so then you start feeling ill so that's the motion sickness and then there's a third thing where people are fine playing the vr stuff but then they get acclimated to it. Oh, yes. And when you take it off. When my producer took it off, he was like, this world is a cartoon and I'm not sure I like it. So it was a little strange. Yeah. And that can persist for an hour or two even. And it's unnerving. That's what I experienced

with head mount displays is I get acclimated to them. And then when I go back into the real world, I feel a little odd and maybe even nauseous. And so getting back to our stuff, what I was trying to work on was a head mount display. And I was trying different experiments to figure out why these different things happen. And there's, I had a system, I had a projector and a beam splitter. And the beam splitter was supposed to direct the image to my eye, but I put it in backwards and it was projecting out into the world. And in our lab, we had a piece of this retroreflective material, which is the heart of our system, sitting out there.

And retroreflective material is like road sign material or the kind of silver... The vests? Yeah, strips on silver vests. Bike reflectors. Yes, exactly. So any light that falls on a retroreflector goes directly back in the same direction. So we had a piece of this material in the lab

and I put this beam splitter in backwards so the image was projecting out into the room. And I looked into the system. I'm like, whoa, you know, I see this beautiful image at a distance and I tracked it down to this piece of retroreflector. So I started researching, I'm like, you know, this solves a lot of problems. Like if we just take a display, and we set it out on the table, which is the retroreflector, your eyes are always going to be converging at the right distance,

your focus is going to be at the right distance. And that solves like two of the huge issues. So I started going down that path. And I built some prototypes with some huge projectors on this, pretty much a helmet. And it... Oh, there are pictures of this. There are pictures online and it's like two cell phones is what it looks like. Yeah. In front of your eyes almost. Yeah. It was two laser projectors and weighed a ton i mean it was 10 15 pounds probably i had this wire bundle that was i don't know maybe three inches in diameter and be fun to carry around yeah and so i started showing people around valve like hey put this on and you know the first thing out of their mouth was like wow but you know no one's going to wear something ridiculous like that and

that was kind of also the disconnect i'm like no don't worry about that that's my job is to miniaturize it we could do that part later yeah it was kind of funny they would fixate on the cable i'm like don't worry about that we'll do micro coax and they're like what and so we started working on that and we found that we could have these AR and VR experiences that wouldn't cause all the headaches. But we were still using kind of not good tracking. And so as we were starting to develop better tracking, we started realizing that, oh, the nauseous feelings go away and this acclimation thing goes away as your tracking gets better. Because when you start moving your head and you present the user with the correct image, then there's no more conflicts there. That sounds a little magical, but okay. That's very cool. So you have the augmented reality,

which means that you can see through the glasses. And then you have the augmented reality, which means that you can see through the glasses. And then you have the virtual reality, which means you can't. Do these glasses have both or are these both in the, I don't see a solid bit. Okay. So for the listeners, I can describe what it looks like. So there's two clear lenses. They're shutters. They're just like for like these like these flicker um 3d tvs yeah okay so they they go dark and then they go clearish well they're kind of like dark glasses we open up one eye at a time okay but we're actually driving them quite differently so you

don't see any flicker so for instance a television set will have to draw an image and then a shutter will be able to open for only 5% of the time of the frame draw time, because you got to get the image out there first. And then it flickers open for a second, has to close because the next picture is coming down. And then it opened, the other eye opens for a 2 to 5% of the time. So we actually get to keep the shutters open 50% of the time and you can't even see any flicker. And then the two, we have two micro projectors that sit just about eyebrow level on the top frame and those project out and that's projecting 120 Hertz stereo images out. So that's, that's away from me. That would be, if I was wearing them, it would be towards you.

And that would make you different. How does that work? Okay, so I'm projecting the stereo image out into the world. But it's a very dim image because we're not putting very many photons out. Like if I shine these glasses at you and I'm trying to project an image on you, you would see nothing. And to you, it would look like two dim leds at the above my eyebrows and given that it looks a little bit like the the dentist uh glasses that are kind of lenses above glasses you would look like your eyebrows were on fire yeah it's uh we don't win any uh uh non-dorky awards but in but in the world of Google Glass being maybe the most stylish.

None of them win non-dorky awards. Google Glass being maybe the most stylish, and then ours probably comes a pretty close second compared to all the displays out there. So we're projecting these stereo images out there, and when we put this retroflective sheet a large sheet you roll it out onto the table this very dim image that's going out that you can't see on anything gets redirected back to your eyes on the the retroflective sheet and you see this super bright super vivid image because all the photons are coming directly back to you right and and where where if you reflected it on a white doorway you wouldn't see anything because only some photons come back to

you yeah 90 of them are scattering reflective stuff it all like goes right back to you yeah it's like it's cool yeah it is it was and it's just kind of dumb luck running into that and it it solves so many problems their optics optics, like I was explaining, is like removed off the head and it's at a comfortable distance. And then power consumption, like all these near to eye displays are very inefficient. So for our system, we can just pump out very few photons

and the return is very high on those. So we don't have to burn very much power because all these photons are going out and coming back. Wow, that solves so many problems it does the the difference is um you're not going to walk around the world with this you're going to need some of this retroreflective sheeting which is it's like a fabric and you'll have to roll it out when you want to have your experience or if you want to have a giant monitor on your wall, you just hang a poster-sized piece of this. And it's cheap. It is.

This is a material that's been around forever, and it's not as cheap as cotton, but it's not expensive. Pretty darn inexpensive, especially out of China. We've been quoting the material. So it's a couple dollars per square meter in bulk. I don't know what it's going to be in retail, but it's still going to be pretty darn inexpensive. So anywhere the material is, you have a display

because in conjunction with the projectors, once we left Valve, we went away from using this webcam-based tracking system. We developed our own modulated infrared LED system. So on the surface, you put out this little square that has a bunch of infrared LEDs that are modulating out, kind of like Morse code, coding out their IDs. And then on the glasses, we have a FPGA

that's doing a lot of DSP work on this camera image of the infrared LEDs and figuring out your head position and triangulating where you're at around the, around the table. So you can walk 360 degrees around the table. The computer knows exactly what angle, what distance and where you're looking at the table. So it can present you with the accurate visual representation of, of the material that's on the table.

So the material, since it's just reflecting the image back you can be any distance from the material but you'll see 3D objects popping up out of the table or popping up out of the screen just as long as the screen's there you can have as much depth into it as you want as much images coming towards your face so if you have a nice square of it hanging on the wall and you're sitting in front of your computer monitor

with this on the wall, you could have stuff come straight out to your nose or you could have it look like it goes off. When I say stuff, I mean computer-generated graphics so it can look like it's extending off forever. Yeah, I mean that would be 3D. Yeah. And if I looked at the same retroreflective material that you did

and we both had glasses on, we could watch our own shows, right? Exactly. It wouldn't really interfere. No, since all the photons are going back to you, you can have any number of people working off the same surface. So for the instance of board games, each person would be wearing their own glasses, they would be playing, and they would be seeing relevant data for them. And from the

perspective, so for instance, we might render a little character on the table, I would see the backside of the character. And since we're sitting across from each other, you would see the front side of the character. And the computer, since it knows where you're looking can present you with the correct perspective and if we swap places or walked around it would just look like we're walking around this character as if it was a real object on the table which by the way it is a real object on the table in front of us which is why we know which way we are but you know you can't see that so uh so if i wallpaper my room with this i it's pretty much a holodeck star trek was wrong it isn't the red and black it's going to be all silver reflective right yeah well at least as of today um or i should say 10 months from now when we ship our units um yeah we imagine uh i, that was one of the conflicts that we had at Valve.

People looked at this and they're like, well, you know, most of our video games are first person shooters. And how are we going to convince people to hang poster size pieces on the wall to have these really immersive like VR experiences? And so that was a bit of an issue. But as we're finding out now that we're doing Kickstarter, and we Kickstarter and we've actually been to six or seven conferences and we've shown thousands and thousands of people our demos, folks get really excited about it because it's the first time they've had a really comfortable experience that they're not feeling choked off wearing some of these other display devices.

And people are really getting excited about making holodecks. Oh, yeah. I mean. It's pretty extreme. I don't think most people will do that. Well, no, not initially. But, you know, you do it. I would do it in my garage and then I'd invite my friends over.

Or somebody will have the old arcade will come back and it will just be a room that everybody can go play in. Yeah. And it's neat because the photons photons as they project out from your glasses will go straight past things. So if I put my hand up, all the photons will go through my fingers and if I want to render graphics like there was something out farther than my hand, the photons go out, hit the retroreflector, come back through and it looks like that thing is way out there in the distance.

I bet that's better for motion sickness, too. With the Oculus Rift, the knowing that the hand is in front of you, but there is no hand that you can see was a disconnect that was brain hurting. Yeah, yeah. That's very cool. Yeah, I'm pretty proud of it. And since we don't have any complicated optics, it's pretty straightforward stuff. We can drive the cost way down.

And with our Vision chip that we're designing in FPGA, we've actually integrated everything onto one chip. So that makes the entire bill of materials for the glasses extremely low. So when we finally go to production, we're going to take the FPGA, we're going to convert it to a metalized gate array, which is an ASIC. Yes. It's in between full custom ASIC, which is extremely expensive. Yes. Metalized gate arrays have a predefined underlayer, much like an FPGA, but they don't have the routing resources on there. And what you do is you, you buy the metal

layers. And so you have a certain number of multiplies that are predefined, a certain number of block rams and uncommitted gates all over the underlayer. And that's what you get to work with. And then you metalize it, and your design is almost as good as a full custom ASIC. It's low power, it's smaller, and... Cheap. Cheaper, way cheaper. So currently we're using about a $15 FPGA. We're anticipating that to be probably between $2 to $5

in metalized Gatorade. So that takes the cost completely out of the box. Yeah, yeah. That was a trick I learned after i got into doing ship design and product design i started doing toy design and well that was one of the things that kind of well that made you a hero of embedded systems is uh is when you reverse engineered and then designed the entire commodore 64 and video game system on an fpga and then made it cheap enough to sell millions on QVC. Yeah, that was a dream project. I mean, at the time, I hadn't done a project like that until

the AR glasses. There was nothing as magical as that C64 thing. At first, it seemed kind of silly, but then as we put it together and the metalized Gatorade came together and it all worked and we put a ton of love, me and the software folks put love into it and we put Easter eggs into it. It became this really magical project. And we made it hackable so you could open up the toy and the hackers could download their own games or they could use it to control a robot or whatever they wanted to do with it. And it became this runaway hit because the hackers promoted it. It made it to all these news sites and we just sold out like immediately.

It was funny, the toy company was going to sue me. They were threatening to sue me because we put these Easter eggs in there and then part of the Easter eggs explained how to open the toy and modify it, which just frightened them to death. Yeah, if I'd done that at LeapFrog, there would have been some words. Oh, I bet.

But then they sold so many, they decided they didn't care? Oh, yeah, they backed down. I mean, you know, it's been so long, I can tell the true story behind it. So we put all this stuff in there, and we found out that it was a no-no because some of the toy execs like were at the factory and saw me drop into the special mode that showed all this stuff and they forbid us from talking about it and so a friend of mine helped me make a backdated blog that we we pretended it was a Chinese factory worker and we had, you know, broken English blog posts of like hacking things. And like,

I'm working on this new toy and it has the secret menu in it. And so he was like really savvy at putting this together and promoting it to all the hacker sites and seeded the whole thing. And, uh, yeah, we did it. We're like,

you know, they're got to do it. We put all this love into it. I'm a troublemaker. You can see why I got in trouble at Valve Software. You know, it's like, you know, I try to do

what I think is the right thing, and I stand behind it, and it doesn't always sit well. Well, there are some people, I think, who are better at being employees, and some people who are better at not being employees because it's hard to listen to somebody else's vision. And if your vision is clear enough and clearly it is well appreciated by the people you want to be appreciated by, excellent. Yeah.

I didn't know you worked at LeapFrog. That's very interesting uh i'm sure they're very strict since their target demographic is what four to twelve or something uh well uh my own learning leap which was my last big toy there does in fact have an easter egg so uh it happens absolutely i try to put an easter egg into everything i think i've only had one project where it got completely scrubbed of every easter egg just at the last minute they i'd stuck a bunch of stuff in a it was a really boring chip it was just a video compression chip and so i'd stuck a bunch of stuff in the registers, like my name and I don't know, person I was dating at the time or something.

And so our verification team had not picked up on this at one point. And one of them had made a mistake in one of their test vectors. And it started reading out from this area that I didn't expect them to read out from. And they discovered it and made it a bug. And then it had to be removed. Oh, that's so sad. It's sad. No one would have ever seen it but it

stripped me of that yeah piece of dna inside of the and there's just it i get more affection for the things that i work on if i get to put a little bit more of me in there um which is i mean it sounds like when you have such love for your work, it makes it so much better. Yeah, it's my passion. I love doing this stuff. I love it. You know, I love creating products. So I always want to add a little extra.

You might find this story kind of funny. It's kind of along the same lines as a leapfrog type device. So I was working on this toy. It was Bratz. Oh oh the little the the dolls the it's toy for girls that's yeah i mean our internal code name between a few of us that we're working on is called little hooker dolls but yeah that seems very appropriate you know i'm not big on like bling bling and all the stuff that they uh sparkles he sparkles and you know a gameplay that's like your boyfriend's cute what's his name kind of kind of dread hard yeah exactly so

it was kind of fun one of the we were i was developing the hardware and it it came up and the software group was working off remotely and they sent over test software. It was the final drop. But they made a mistake and they sent me all the source code too. So then I recompiled the source code. I had the little brats swearing and belching and farting. So I wrote them back. I'm like, okay, hardware came up great.

It's like 95% there. There's just some audio artifacts. And they're like, what? You send this to us immediately. Like, okay. So I FedExed it down to, you know, where they were at. And they're like, damn you, Cherry. I kind of felt bad afterwards

because this was going to be their final code drop and they had got the whole team to agree to come in on the weekend and work on it. So I kind of disrupted some folks' weekends. I can totally imagine being the software engineer on that. I don't think I've ever done another project for them to tell you the truth. It was like when we got the uh the naughty word

list printed on t-shirts because in leapfrog toys if you push you know certain letters together it would sound out uh the word some of which it instead says that's a neat set. Or, huh, those letters are funny. And instead of sounding out whatever it was you intended. Yeah. Nice. Getting those all printed on the shirt was a politically limiting move. Oh, really? Well, it wasn't. I mean, just because they never published what the whole.

I mean, then, you know then you might learn some new words. But back to CastAR, we've kind of mentioned Kickstarter. Why did you do that instead of the VC route, the traditional way? So we considered both and for quite a while. So we left Valve, we developed all the new tech, We got it working really well. And so we considered VCs, but you have to give up a lot of equity in your company to do that. Oh, yeah. Which, you know, there's no strings attached, kind of,

when you go to VCs, which is a plus. They're only looking for one out of ten companies to be a success. If you're not a success, there's no one breathing down your back. Conversely, Kickstarter, it's a crowdfunding site where we go to the general public and we say, here's our prototype. It works. We're demonstrating it.

People have used it. Believe in us that we can actually get this to manufacture. And they start putting their money into the project in return they get rewards back so some of the rewards in our thing is you get the glasses and or you get the rfid grid or you get the virtual reality clip-ons and so the risk in that is if we botched our bill of materials or disaster strikes, there's a epoxy factory fire. Remember when that happened a few years ago where the prices of chips went up by like 80% because of lack of epoxy of all things?

So all that is a huge risk. And now we're accountable to tens of thousands of people that expect to get a product. So that was the decisions that we had to make. But you like shipping products. So this string doesn't seem as arduous as my idea now is 50% owned by a guy who did nothing for it. Exactly.

Give me a bunch of money, which is nice, but it would have been better if guy who did nothing for it. Exactly. Give me a bunch of money, which is nice, but it would have been better if he'd been working for it. Exactly. So, you know, I feel very confident. I've shipped lots of products and, you know, there's still a small chance that a disaster will strike, but we can actually minimize that. So we did a lot of that up front.

We worked on the bill materials. We contacted vendors. We of that up front. We worked on the bill materials, we contacted vendors, we negotiated prices up front, which is extremely difficult to do so early. Oh yeah. And you've already contacted Chinese manufacturers and you're getting the whole plan set up. Exactly. Logistics. Which is tough if you are a small company and new. You don't have the backing to say, oh, sure, we can totally do this. And your manufacturer may wander away if they get bored. Exactly. And, you know, they may not have capacity by the time you actually get around to manufacturing,

going this far in advance with no money to put down and say, like, I want this capacity and lock down these benches for us. So, yeah, there's some risks there. We looked at it as, all right, people love it. We've shown it. It delivers on what we say it delivers on. And we believe that people will fund us. And if something goes horribly wrong, there's still VCs out there that are circling like sharks.

I mean, as we showed it more and more, more and more VCs were approaching us. Even right now, we're in the middle of Kickstarter. We're almost to our funding goal. And it's been a day and you're at 332,000 out of 400 is your goal. And you have 30 days to go and 1,351 backers. It looks like you're going to make your target. Yeah, I think we're going to make it. And this 400,000 that we asked for, that would be me and Rick doing everything for free

and going and manufacturing the few hundred units. So we're hoping for more than that. Do you have goodies planned for going over? Oh, we do. Yeah, we actually, we launched it and we're like, okay, maybe it'll be a week and we'll get to the 400,000. And it happened almost overnight. We're like 18 hours into it and we're almost to our goal. So last night we sat down like, okay, we have some ideas of some extra goodies we can add. For instance, if we were to get $600,000,

now we have a little bit more budget. We can hire an engineer to help us ship it with more games. So last night we made some more tough decisions. We had to sit there and crunch the numbers. We pulled up our spreadsheet that has everything, and like, okay, we're going to put an engineer on this project for four months, and it's going to cost us this much, and we think it's going to by spending this money out of out of the out of your lunch money yeah you know we could put that in our pocket but we can spend it and we can we can make the leap and

we think it's going to drive us beyond the two hundred thousand dollars more at the stretch goal so you know for instance one of the things, we're going to ship another game with it when we reach $600,000. And then when we reach $800,000, we're going to add a microphone and a headphone jack to it. So you can do positional audio. This is all stuff we've tried,

but we had to cut it to make this really attractive $400,000 price point for funding. And as we get more and more money, we can start adding the features back in that would make it the ideal project. And that'll generate, in theory, excitement with the fans. They'll see, like, all right, we hit $600,000. Now we're going to get this Dungeons & Dragons remote and local dungeon builder game. So you can do board games remotely with your friends and that'll drive excitement and they'll try to bring their friends in.

And yeah, well, I mean, word of mouth is, is big. That's really Kickstarter-esque is, is making it word of mouth. So, so which, which pledge level should I go for? It looked like the first one that I got everything that I needed to really try it out was the starter package at $189. Yeah, so we have a $189 package, which is the Surface and the glasses. And then you provide your own input. Might be keyboard, might be mouse, Xbox controller. Then we have another level. The 285 is the Pro Package, which is kind of what I have my eye on,

especially if I can somehow write this off as a business. Oh, there you go. Yeah, the Pro Package. You get the glasses, you get the surface, you get the magic wand. Which we haven't talked about yet. There's so many things that we developed for this. Yeah, you do have a lot of peripherals already ready to go. We've leveraged a lot of the tech

and we've twisted it around in different ways that we can make interesting peripherals. For instance, this ProPak has a magic wand in it. And the magic wand is the same head tracking camera as the glasses. So we have some millimeter accuracy on your head position. So if we put this on the end of a wand, now we have some millimeter accuracy on a wand that you can move through this 3D volume that you're rendering. So if you want to reach out and touch your character in 3D space, we know exactly where your hand's going with this wand,

and it's got a trigger and a little analog stick. And so we've already done the work so we're we're taking some of these things and moving them forward to make interesting peripherals to make the the the package even more desirable so so the magic wand is it going to be a lightsaber can i have a light yeah actually that's so cool that's one of the things we're thinking about doing for a demo. Like this is one of, you know, part of a Kickstarter campaign. Like I've, I'm not much of a PR person and there's so much psychology and PR involved with the Kickstarter. It's, it's so much, it's like a game unto itself. It is.

It feels a little sleazy and I apologize for folks that have to endure my... I feel bad about doing that kind of stuff. About doing Kickstarter? Or about doing the PR? PR. You know, again, there's divides between PR people and the hardware people. We poo-poo them as the... Sales bots.

Yes, sales bots that are pie in the sky and don't understand hardware. So there's some of these games we have to play. And part of the Kickstarter campaign is constant updates. And so we've lined out these different things that we're going to do throughout the month to show. Like lightsaber with the magic wand is one of the things we want to... We don't know what it's going to be yet until we actually code it up. But we have a Jenga block game that has Jenga blocks.

We were thinking about converting that to a lightsaber because we've shown the Jenga blocks quite a bit, but we thought it would be kind of funny if you took the lightsaber and just vaporized the, and so it would be the, you'd have the wand, which is the handle, and then you would have virtual, you know, the lightsaber part of it that would be rendered graphics. And as you swing your hand through the scene

that has 3D Jenga blocks sitting on your table, you'd... And it would vaporize them. Oh, that would be very fun. It would be a great way for us to show the orientation of the wand. That's something that hasn't really come off in any of our videos that we've done so far, is that you can do full orientation and position yeah so you've got a little imu in there

well it's it's all absolute positioning off of the the image tracking oh oh that makes sense yes so um we are considering putting an imu in there and that might be one of our stretch goals i mean there are times like if you want in there, and that might be one of our stretch goals. I mean, there are times like if you want to take it and swing it behind your head, it's not going to see. The Indiana Jones whip action. Yeah, but again, it's some of these trade-offs. We would have wanted to just put a full IMU in there from the start, but we had to make it attractive.

And they get expensive, especially the good ones. Adding magnetometers to them. Yeah, and then the magnetometers. And the accelerometers have been around forever, but the gyros are just in this sudden shift where they're going from $15 for a 3-axis to 50 cents for a 3-axis. And I'm ready for that to be done

and then to ship all of the little cheap ones so that I can put them on everything. Exactly. Oh, well, all right. Well, I'm waiting for them to be at the nickel level so I can integrate them into

toys. Yeah. I mean, that's in the toy industry and you know, this working for leapfrog, but maybe they, I would imagine leapfrog had higher margins than a lot of the toys I worked on. For instance, like this Commodore joystick thing we were talking about, for leapfrog but maybe they i imagine leapfrog had higher margins than a lot of the toys i worked on

for instance like this commodore joystick thing we were talking about we had margins of two dollars for electronics circuit board wires everything it's like you can't do much at all so you have to constantly keep your eye open for new technology that was 15 now is a nickel for some reason like it got integrated into cell phones yes all the Like it got integrated into cell phones. Yes. All the things that got integrated into cell phones just got better and better. And, well, I mean, that's part of the magic of the CastAR system is our initial tests with tracking,

the head tracking, we were using these $50 image sensors that were global shutter. They would snap a picture all simultaneously. All the pixels would get exposed simultaneously. And then we'd dump those out and do image processing on them. And that way you could have a lot of motion in your head or whatever, and you would have an accurate lock. And I really wanted to use the $1 image

sensors out of cell phones, because there's just no way we could ship a product with a $50 image sensor. So I started looking at different optical tricks and I found this pretty novel little optical trick so that I could use a rolling image sensor that's not snapping all the pixels at once. It's just snapping one scan line at a time and use it to track with the same precision and actually faster than a global image sensor. And so that was that moment, just like with the toys, where there's been this shift, and another little shift in optics allowed us to combine those two, and now we have this really magical project.

The tipping points. Yeah. Yeah, they're very, and they just come faster and faster. Very cool. I'm sorry, I keep taking us off track. But some of the other things like the Pro Kit. I like that you think that there's a track here. But okay. I can ramble forever and take us down.

We'll be talking about race cars again pretty soon. Hey, I have race car vehicle dynamics. It's a really great book. It's really big and thick. I also have done race cars. Oh, that's awesome. Okay, well really big and thick i also worked on race cars so i'm there for you oh that's awesome okay well we'll have to go back to race cars because i've got stories and talking about being a troublemaker but um just for completeness for the the listeners um so since i was working on all this near to eye display stuff and i understand it very well

we went ahead and this was part of our secret project. We'd been showing people the projected augmented reality, this thing with the retroflector. We've been showing that for six months and we know it's a good experience. But there was this resistance to a system that was only projected augmented reality. So I built a free form optic system that clips onto the glasses and it takes the projectors. Oh, you mentioned the clip-ons. Yeah. So here's a mock-up of one here. It actually has the reflectors in the spot, except for the one fell out there in the top. So you project out, it goes into a first reflector, hits that reflector, goes downwards towards a secondary 50% beam splitter. And then that goes forwards to a free-form optics

that redirects it all back to your eye. And that allows you to clip these onto your glasses. Then you can have the VR, classic VR, true VR, as we're calling it, experience, like Oculus. Or the reflector on the front is 50% reflectivity. So we have a little cover that's going to go over the front. You can remove that and then you can do true augmented reality. Now, that's great.

It still has the focus issues and stuff. So it's for people that are accustomed to that type of display. I think I'm more excited about the way you're doing augmented reality with being able to see my hands and put all of that in there. I hope you're thinking about haptic feedback at some point. Oh, I wish I could tell you about all the haptic stuff we did at Valve. Oh my God, it's probably all dead. That's sad. It will come back. We'll need this. I've read Ready Player One. We're going to get all this really soon. There is some really cool stuff out there. Yeah, I probably can't talk about that stuff.

Fair enough. Just tell me you're thinking about it and that you thought it was cool in the past. Yes, so haptics, we know there's magical experiences there, especially combined with VR and AR. It was beyond the scope of what we could do in a one-year time frame. So that's the VR clip-ons, which are a low-cost add-on. That's part of the Pro Kit.

So that's the entire Pro Kit. You get the wand, you get the glasses. So VR clip-ons have no electronics. They're just optics? Yeah, and a relay optic to bring the tracking camera forward. Yeah, so it's a passive device that clips on there like that. I mean, that's very cool. VR for just optics.

Yeah, so that was a very low-cost thing that we could add and add two more functionalities to the system. And a couple of checkboxes. Yeah, yeah. So now we can do do it all which is cool um the one thing i forgot to mention about the glasses that i'm very proud of is we've cut all the weight out of them so the initial head crab that i was talking about this giant thing weighed 15 pounds and i continued to shrink it shrink shrink it, shrink it. We made a standard def version of the glasses.

It's about 300 grams. And then for final production, we've done some prototypes of putting the entire optic system together, which we've taken all the glass optics out and we've replaced it with hollow space kind of optics with thin film beam splitters and homogenizers and stuff like that.

And we've got the weight down where it'll definitely fall below 100 grams. And so to put that in perspective... That's a pair of sunglasses. Exactly. Yeah, a pair of sunglasses might be like 50 grams if it's kind of a big, heavy pair of sunglasses. So our system is going to be somewhere between that 50 grams and 100 grams. But it's going to have a cable for batteries. It has to have a cable, unfortunately. There's no solution out there, at least that I've seen on the horizon,

that will allow us to do wireless video that we can actually pack a battery on your head and deal with the thermals. For instance, there's some really good wireless HDMI transmitters that can do enough bandwidth to transmit video to the glasses. But you'd only be able to play one person in the room at a time because it consumes all the bandwidth around 5 gigahertz. So you can't have multiple channels. So it would just be one person experience. And the unfortunate part is the receiver is 5 watts.

So how do you dissipate 5 watts on your head? It's difficult very warm yeah so we have a talking about the cable um listeners probably geek out on this we're using micro coax and high speed serializers so we how high speed is high speed well it's in the gigabits so That's really high speed. So the system, the way it works, and your folks will probably be very interested in this, we get some retimers,

so that take the differential pair and retime it. And then we send it up dual micro coax for each channel. And it goes up to the receiver IC on the glasses. And then it gets deserialized and then goes out to the displays in parallel. But that allows us to do a cable. So as you can see, the cable here is, it might be a little thicker than a pair of earbud cables. Yeah, I'm comparing it with my earbud cables

and it is a little thicker, but it's not a lot thicker. Yeah, so our idea is with the hardware tracking, we'll be able to hook to cell phones eventually. We're actually doing our experiments with Android now. And you'll have this micro core. This has a 64-bit processor. This should all work really well together. Oh, sure.

Yeah, as far as rendering graphics, the cell phones these days can do it. It's not a problem. Up to this point, before we did the hardware image detection, it would have been impossible because you would, before we were transferring entire images to the PC, doing image processing, burning a lot of time doing that actually was added two frames of latency to the system. And then there's just no transports way to do that on phones. And actually what we're thinking about doing in our initial test,

looks like it works, is we're going to use the earphones because we need a low latency deterministic interface to where once the tracking system sees these infrared LEDs that you've put out onto the surface, it figures out its position, just sends that down. So it's only about 80 bytes max for all the input. How is that over earphones? I missed that.

Sorry. Oh, microphone part of the earphone jack. Oh, okay. I'm sorry. I should have said microphone jack. So yeah, it's so low bitrate that we can just push it over there. And it's deterministic. So we know the latency.

We have a lot of prediction in our head tracker too, so as soon as you start moving your head, we start predicting where you're going, and actually with pretty high precision, we know you're going to have this certain velocity, and we kind of know where you're going to land, so we can render the frames ahead of time, and even though, best case on a computer,

it's going to be two frames of latency, because there's a front buffer that gets in a back buffer, you know, frames get written into one and then they do this frame flip. So there's this two frame latency. So we predict through that and that gives us the ability to keep things to look like they're locked on the table. This is all so very cool, and I want to keep you here to answer all of my questions, and I want to hear more about FPGA and how you're doing the image detection,

but we are almost out of time. See, I told you I could just ramble until... You're not the only one. I should mention, okay, I need to get the pitch in here for the rfid grid before the last component of our system is a grid of rfid reading coils green thing yeah so this looks like a giant circuit board circuit board with little boxes on it yeah so this is our prototype it's not what we're going to ship we're doing some experiments with flex circuits now to make we we want it to be where it can be rolled up we don't flex circuits are just i love them and i hate them at the same time yeah they're so awesome to be able to roll things up and and to

have it will it last curve but then as soon as you curve it and uncurve it and then curve it again it's broken yeah that's the that's our anxiety we're doing some testing now um we're not 100 committing to that's what's what's going to be like but this rfid grid allows you to put passive rfid tags down there's just these high tags that we're using right now and we can do centimeter tracking of their position so you can put them on the bottom of your dnd figures and put them down and we know where they are on the table. And it's your co-founder Rick who just loves D&D. Oh gosh, he's a D&D nut. His place is full of figurines and stuff. I know nothing about it. I can appreciate the coolness factor of it and I developed all the hardware to make it happen, but he really digs it. We've also developed our own RFID bases that are bi-directional. So we have communication

out to a microcontroller. So we can do a lot of cool things through the RFID grid that goes underneath. Like you can move your figurine to different grid spots as though it were a board game, like we were playing Life from, you know, 1984 or whatever. But it does a lot more than that, right? Yes. So there's bi-directional communications. We have a microcontroller on there and GPIOs that you can talk to those GPIO and turn them on and off. For instance, we could put this on a remote control car, a physical remote control car,

turn motors on and off, drive the remote control car around your table and drive it. And the computer would know exactly where the car is and keep it from falling off the edge of the table. Maybe you're driving physical cars in a demolition derby and, you know, they run into virtual things. There's a lot of fun things you can do that way. Plus, we can also add buttons. So we can have props that you could put down that have buttons, like the demo that I'm holding here in front of you is a little miniature, and it has a button, and we use that as input.

So you could have an array of buttons on the bases. So I could put him down, and there might be a button for defense. And I push that button, now he's in defense mode. And there might be another button that I push, now he's in fight mode. But you can have LEDs too. So when he died, he could turn red and or you can render graphics around him too oh that's true because you do already control the visual don't you um but we even talked about like even the most extreme silly it's like if you had a dragon you could put one of these little like um smoke generator things

and yeah lighter that would be awesome. I'm talking about blowing things up. That would be hard to get through certifications, I think. It's for entertainment purposes only. Don't use actual butane. By the way, if you want to use actual butane, it works. So to summarize the tracking system, so the flex circuit is something we're looking at we're going to see if it'll hold up but if it doesn't then it'll be little squares which will probably be you know 40 centimeters on a side and they'll click together and you can

configure this rfid grid in different configurations and then you roll out the retroreflective material on top of it and then you can play your very immersive board and dnd and warhammer and all these various settlers of katan yeah physical prop things you know blend that with your magic wand you can do other things into that 3d space well there's so much I can imagine doing. I mean, board games, D&D, I can see how that would be a good place to start this market. But then my dog went to the vet this morning.

They showed me an x-ray and now I want to be able to get the CT scan where they do an x-ray slice, a whole bunch of x-ray slices. We've actually talked about doing that and see if we can figure it out. I could be part of Inners the, the, where they do an x-ray slice, a whole bunch of x-ray slices. We've actually talked about doing that and see if we can figure it out. I could be part of inner space, the movie and wander around my dog's interior. That would enjoy that.

So we've actually looked at that data set to see if it's possible. It's complicated. There's, it's billions of points that would have to be rendered but there might be some tricks we're working with a shark researcher um out of stanford and they tag sharks and turtles and other sea life and they track their position around the ocean and their dive and their position but they've always struggled with visualizing this and how to convey it to their colleagues so this gentleman came to our maker Faire booth, saw this, and he's like, this is the tool that I've been waiting for for ages. Now I can give my colleagues a pair of glasses.

I can wear the glasses. We use the magic wand. We roll out the surface on the table. We can walk around the table, and we can look at different perspectives of the ocean floor with lines that are rendered at the dive depth of where the sharks are. And then you can reach in with the wand and zoom and scroll with lines that are rendered at the dive depth of where the sharks are. And then you can reach in with the wand and zoom and scroll and click on different paths and analyze it together.

And it's a collaborative tool to visualize these complex 3D data sets. All these collaborative, I mean, you said that, and I went back to one of the problems having to work with memory maps. I'm like, if I could have everybody visualize where the RAM and ROM are going, then they would know. And they would stop saying, can we add more buffers?

Oh my God, I have a story about this. I made a visualizer for a memory system on this embedded processor or embedded SOC I was building. We had some leak and something was running wild. So the way I debugged is I had a video port coming out of the thing. So I just mapped all the memory so I could just see it on a VGA monitor. And you could see a pixel changed when, when the processor went off in the weeds and wrote to the wrong spot. This sort of visualization would be so cool.

Medical, environmental, weather. I mean, to see clouds from all the way through. Can you imagine being able to see what it would be like in the middle of a tornado? Yeah, yeah. Well, we did some experiments at Valve where we were moving these, these hemisphere cameras, they record an entire hemisphere down the hall.

And then you could wear VR and you were tracking your head position. So as someone dollied this camera down, you could look around at the walls and the different areas. And it was pretty disorienting, but it was pretty cool too. We've been talking to a big airplane company about using a souped-up version of our glasses to do flight simulators. So they have millions of dollars invested in these projector systems where they project onto these special parabolic reflectors and go out into the windows. And it's very expensive for them. And they saw this and they see potential of putting retroreflector into the windows of the flight simulators and making the entire system much smaller and they can even for the instrument clusters they can put retroreflector

in there and since you know where everything is you can project each pilot is wearing glasses and will project the correct perspective to the window and to the instrument cluster and it eliminates this complicated multi-projector system that costs millions of dollars. Now for maybe a souped-up version of the glasses, that's $10,000 and has 4,000 pixels per eye. Which is so much cheaper than the simulator is now. More portable. I'm sure that the military will be,

if they haven't already knocked on your door, will be knocking soon because they have a lot of the same problems i'll have to let someone else deal with that i'm not a fan i end up working on military stuff i'm like this isn't what i meant yeah i've worked on darpa projects too like darpa is like you know there's always milestones that are almost unachievable and you can't get that next piece of money because it's just on the edge of unachievable everybody thinks that working for the government means that there's so much money but it's not it's not free it's really really not free worse than vc there there are easier ways to make money. All right.

But that's off topic. And I am really excited to see this and to learn more about it. It's cooler than I knew. I want to totally play with it. And I think, yes, I do need one. Anything you'd like to leave us with? I'm sorry we didn't get as technical as i'd hoped but there's so much to cover well we can do that another time i mean there's so much on the guts of the the image tracker that um i'm super proud of and i could just go on for another

hour like there's an eight wave pixel pipeline in there that's all fully custom. And it actually has a 6502 inside of it. Anyway, um, the, um, our website's technical illusions.com. The product's called cast AR. Um,

our Kickstarter is currently running. I don't know when this is going to air, but, uh, check it out. So it's running. You'll be on day two. Uh,

it's running through the end of October. Oh no, it's running through the'll be on day two. It's running through the end of October. Oh, no, it's running through mid-November. Yeah. We tried to get it done before Thanksgiving. One thing we're very proud about is our video was completely shot with CastAR technology. We had the option of compositing 3D fantastic looking stuff in there. And we decided that even though we can only capture 10% of what the experience is really like, we're

going to rig this to a camera. We're going to shoot everything with CastAR. And you can see it. They're playing a game and it's 3D and it's fun. It looked great. Yeah, I think that's all I have.

Alright, well there will be links on the show notes. So if you want to see the Kickstarter, you want to see Technical Illusions, you can have a link. Although, really, Google, not that hard. You've been listening to me speak with Jerry Ellsworth about CastAR, their new augmented reality and virtual reality and all sorts of reality sort tool. Thank you very much for joining

me. Thank you. And thank you all for listening. Many thanks to my producer, Christopher White. He wasn't here to keep me on track today. I hope you don't mind, but he will be editing. So we'll see how it turns out. The show notes will have links, and there will also be a link to the contact button. If you want to say hello or ask a question or leave a comment, I'm happy to forward it along to Jerry. Or you can reach her on Twitter. And check out embedded.fm. That's where all this will be. That's the show for the week.

I have a quote from Albert Einstein to leave you with. Reality is merely an illusion, albeit a very persistent one.