a16z Podcast - a16z Podcast: The Present State and Future Possibility of Virtual Reality

Episode Date: December 17, 2015

Our virtual reality-enabled future is arriving, but it’s hard to know -- as it is with every new technology platform -- how quickly we’ll all make the transition to VR and what it will ultimately look like. For example, beyond gaming and entertainment what applications does it seem like VR is best suited for, and will we all be wearing full VR-enabled body suits some day? When will the Matrix become real? a16z’s virtual reality-obsessed Kyle Russell and Sakunthala Pandit are joined by perceptual and computational neuroscience expert Beau Cronin, to offer their informed opinions on the present state and future of VR. They give the rationale for travel, shopping, and education getting the VR treatment, and why software developers would kill for a VR-enabled dev environment. So strap your “face toy” on and dive into virtual reality. The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments and certain publicly traded cryptocurrencies/ digital assets for which the issuer has not provided permission for a16z to disclose publicly) is available at https://a16z.com/investments/. Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.

The content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. For more details, please see A16Z.com slash disclosures. Hi, this is the A16Z podcast. I'm Kyle Russell, a deal partner here at the firm looking at virtual reality and augmented reality, among a few other things. I'm joined today by two people who know a lot about VR. I'll let them introduce themselves. so you can put a name to a voice. Salku, if you'd like to start off? Hey, I'm Sarkandala, Panther Ratner. I'm a research, I'm on the deal in research team here in Driesen Horowitz. I focus on VR and amongst other emerging technologies.

Before this, I was an intern at Oculus. And hi, my name is Bo Kronin. I have a background in computer science and perceptual and computational neuroscience, and I've become particularly interested in VR recently because it allows us to, in many ways, expand our perception of both the real world and worlds that we get to make up from scratch. So thanks you both for joining today. I'm really excited to talk about, you know, kind of where VR is at

and, you know, what aspects of it, I think, are maybe underconsidered by people who just kind of look at it as a raw set of technologies of things that track motion and our screens attached to someone's face. I think that there's a lot of aspects where, you know, if you look at mobile or desktop computing and other ways that we use technologies. Aspects that are commonly discussed aren't even brought up sometimes when you're looking at VR. So one of the things I'd like to kind of just check in on first is where VR is at today in terms of the technology and in the commercial space. So recently, Samsung and Oculus released the Gear VR. Saku, you actually used to work at Oculus, so maybe it'd be fun for you to talk about where the consumer platform is at today.

Right. So the big thing with Oculus is that the VR headsets, which had been around for maybe 10, 15 years before that, finally became affordable and also usable. So we got the right ratio of being VR high quality enough and also it's reasonable for the average person to buy one. So the Oculus Kickstarter came out in 2012 was a massive success. And then we've had two releases so far, the DK1, which is the Kickstarter, the DK2, which is like a second developer kit that Oculus. release and we're still waiting for the consumer which is like the super high quality version which is coming out next year and then to kind of give an idea of the evolution of this tech so dk1 basically a screen with motion sensors that you strapped to your face and that gave you know something that felt like presence then with dk2 they took that same kind of core concept and added a camera that tracked your position actually that's not quite true um so dk1 in like what we say with presence is it's like a buzzword in the VR to describe the visceral feeling of feeling like you're actually in the place that the screen says you are. So the way I explain to people

is if you're in a VR universe, you're standing on a ledge, you feel kind of scared that you're going to fall off. And it's not like a psychological thing. It's like, you know, physical reaction to the distance. Right. Your lizard brain reacting to input. Exactly. And DK1 was not quite there. I don't know. Look at all the videos of people who would like go on a roller coaster and fall out of their seat. Well, okay, even at Oculus v. said it wasn't, okay, maybe, you know, it's, it was, it was, it definitely did hit your lizard brain, but maybe it didn't quite click presence. But with DK2 and CV1, we definitely get to that point.

And I think one of the interesting things about each system that's come out, DK1, DK2, now the gear VR, which we'll probably get to in a second, and then these prototype systems that we will also talk about that are going to be commercially available, is that each one has kind of proven something additional about what this technology can do. I think the DK1 for a lot of people, I've actually never used one, but many people who have, was the first time that they went, oh, my gosh, you can do really high quality tracking. You know, didn't have positional tracking, but you can do really high quality, low latency tracking. The screen is good enough that I can imagine where it's going to go, and it allowed the right kind of people to sort of have that leap where they said, oh, wow, this is going to be real.

It's worth me devoting my time and energy to it. DK2 was beyond that. Now, Sam, with the Gear VR, you have something that's available for anyone to buy for $100 on top of the price of a Samsung phone. That proves that this is something that people can enjoy in their own homes. And so it's just each of these systems sort of taking it one step beyond bringing this to a broader and broader audience and making them realize, oh, wow, this really is for real. I think one thing to add is that the different types of tracking we have in VR. So the DK1 had rotational tracking. So it would track, you know, if you turned your head it would be able to tell which direction it had turned.

But the DK2 had positional tracking, which was like an outside camera. So even if you were walking around and it was no longer anything to do with something stuck to your head, it could actually tell. It was like a more complex measurement of movement. Right. The extent to which DK.1 felt accurate was to the point where Oculus could make an accurate neck model, essentially. When you're moving your head around, did it kind of feel like it was falling your head in a natural way? Then with DK2, you could actually lean around and look at something from different angles, and it was like you were, that's kind of what gave the I'm here feeling of. I see this really cool, I don't know, let's say, you know, the demos we see are cars

or you're in a video game that's an arcade-style shooter or something, and you're in a cockpit and you can go up and read the dials on the, like, mech you're in, where it's, I feel like I'm there because I can move around. And this is actually a pretty good time for me to give my pitch for the power of motion parallax, right? because the ability to do that doesn't, I mean, there's a reason that that is so visceral makes things feel so real is because that turns out to be an incredibly powerful cue for our brains to understand the structure of the environment that we have around us. So when you look at something like the Gear VR, which does not have positional tracking and positional tracking is not available in any purely mobile system that doesn't have an external camera, it's taken a step back there. It doesn't have that positional tracking. And so when you go and you look at it, you'll find that a lot of the content on there, much of which is awesome and great and totally worth the price of admission, is designed around having experiences that are far enough away from you that that lack of positional tracking isn't as obvious, basically because the angles involved are so much smaller that it's not as important to have motion parallax, which is really important for what we call sort of near field VR, things that are in your personal zone and just really, to my mind, kind of ratchets up the level of experience you can provide a whole,

you know, a whole other level. Right, because it's interesting because I think that a lot of people who are real VR fanatics think positional tracking is the end-all be-all. Yes, me. No, exactly. So, but when I played with the gear VR, I did notice that so many of the other aspects had improved, the resolution to the point where I could see pixels, but it wasn't particularly distracting. Frame rates were kept high enough, plus, and this is kind of nerdy, but Samsung uses low persistence display, so pixels are lit up for shorter amounts of time, so you have, like, less blur as you move your eyes across the screen. All these little things added up to an experience that really does feel like worth the $99, plus maybe the phone, if you're like me and refuse to give up your iPhone.

But at the same time, I then go and do, and maybe now we can talk about the future a little bit, a Oculus CV1, which is the upcoming consumer model, or the HCC Vive kit, which is from HCC and Valve. You go and try those, and you can walk around, and you have controllers that track your hands. And that's an even, it feels like a whole other generational shift where it's, how could we go back to the previous way of doing things. Yeah, that's right. And so I think it's worth saying there is an incredible amount that they got right with the gear VR. It's awesome.

I mean, so the standard line that I've seen online, which I agree with is, if you already have one of those phones, just go buy it. Don't think about it. Go get it. It's a no-brainer. And I totally agree. Then thinking about the systems like that you just mentioned, the VEVE or sort of the upcoming version of the Oculus, yeah, there is absolutely a whole other level. So there's two pieces of that that I think are really important. One is, as you just mentioned, the ability to move around in larger volumes, and that's great.

The other one, of course, is input. And being able to use for both of those systems, and in addition, also the Sony system that's coming up as well, your hands in a very natural way without really having to think about it and have one-to-one highly accurate hand-tracking using handheld controllers in those environments. And that opens up not just a whole lot of awesome game mechanics and so on. so forth, but it really starts the race, I think, for people who want to develop other kinds of applications in VR and want to think about this as an overall computing platform and think, you know, like what kinds of experiences can we provide there in terms of productivity apps, just as one example. It's worth noting that the original idea of VR was not just a headset, it was a headset

and gloves. So if you guys remember, like, Nintendo Virtual Boy, which is very 80s, very gorky and retro, it wasn't just something you put in your face, but it also came with gloves. So it's always been part of the idea of VR. It's not just something, not just something that you see, but something that you interact with. Don't you miss when gadgets came with names like Power Glove? I really miss those days where it was like, no, now you're playing with power. Okay, cool.

But no, it is interesting to think that, you know, in a few years, we're, you know, people like us who are really into the cutting edge are going to have what, gloves plus hand controllers, plus maybe cameras tracking our hands. we're going to have all of these different constraints that are going to limit what kinds of software is made and also introducing other opportunities. So I think maybe now, as we're thinking about the future, it's a good time to think about what does design look like for VR in terms of these mobile-based headsets where you're plugging in your phone. They don't have positional tracking, but they can render a 3D world with crazy high frame rates and it looks great. All the way up to these controllers where it feels like sometimes you forget that you're in a virtual reality because moving your hands feels so natural interacting with. the world feels so natural, especially as we add things like haptic feedback. But thinking about those constraints and the spectrum of what VR looks like, what is development for that going to look

like? I mean, are people going to, is it going to be very like differentiated based on different categories? And how does that affect like what gets developed first, you know? Yeah, absolutely. So I'm not a designer, but I am following closely a number of very, very smart designers who are trying to figure out exactly these questions. What does interaction look like? What are the affordances that we want to offer? How can we make these interfaces at the same time, you know, at once powerful and very natural? So I think in VR design, there's a split between, like, realistic design, or skeuomorphic design, and then something which is a little bit more abstract and a little more symbolic. So, you know, with the phone, you know, our buttons, like, everything seems to

kind of make an attempt to resemble a real-world object in a skemorphic design. With VR, maybe a general purpose app, you know, it might be, you know, for your writing app, it might be a virtual desk and you're sitting there writing. That might be the way to do it. Or it might be that there's some kind of more abstract way of interacting with words in VR that we just haven't thought of yet. And it's, it's worth remembering that these things take a long time to develop, like with film, You know, it took like 30 years before the jump cut was invented. So that's like, you know, you're cutting from, you know, it used to be just like filming people on stage like a recorded play.

And it took a while for all those tropes of editing to develop. So film is one great analogy that we can use. Another one that's more recent is the transition to, you know, multi-touch displays on mobile. So if you go back to, you know, the initial iPhone, release in 2007 and sort of, you know, where we, where we started there and where we've ended up now, we as, you know, number one, as a design community, but really as a larger society have absorbed a set of conventions that, you know, back in 2007, no one, not even the designers

knew really that pinch and zoom and, you know, various kinds of swiping were going to become these both dominant and natural and, you know, almost totally subconscious at this point, you know, components of those interfaces. So right now, there are a lot of very smart young designers who are trying to explore this space and figure out, you know, what are the equivalence in VR. I mean, one of the challenges that they face is that the major systems that are coming out aren't exactly equivalent. So they don't really know what they're going to be able to count on between sort of the Oculus Touch controllers, the HTC Valve-Vive lighthouse controllers, which is a wonderful mouthful, and then the PlayStation Move controllers.

all have, you know, some pretty important differences between them. I also think it's interesting to look at the way that the development of the software and then how it's received is going to shape the development of the underlying hardware and technology in terms of right now it's slot your phone into something and maybe have like the ability to tap and use gaze-based controls. And then you have the more advanced headsets, which are basically just screen and motion sensors and maybe extra controllers that you plug into something with a lot of raw CPU and GPU horsepower, like a gaming console or a gaming PC. But as these things evolve, you can see how it could evolve to be the phone still slots

into a headset, but then when you're at home, you plug then that headset into something with all of that horsepower, and it's kind of a almost like surface-like the device changes to meet the needs. Or you can also see a standalone headset that has its own brains that's separate from your phone because it turns out people maybe also want to use the phone for. some kind of input, and then, again, that hooks up to raw horsepower, or, you know, that model kind of goes away and the mobile-based approach just takes off even further. And, you know, once they're as powerful as a gaming PC of today, you don't really mind the jump in graphics

compared to, like, what your desktop could provide. And there are certainly people in the field who say, look, the future of VR is definitely mobile. It's just a matter of. And Paul Merlucky may even be on record saying us, I'm not sure, that the tethered desktop version is a very important, but it's a stepping stone, really, until we can get to that fully mobile version. I think one other related point I would make is right now these systems and these Harvard platforms are all coming out. And it's wonderful that we have gaming and entertainment as sort of this wedge application that is driving the initial investment. That sort of guarantees there will be some minimum viable market for these things because there's a certain subset of people who on day one will shell out and are waiting for this to happen. But then, of course, there's this question, which is, what does the larger market want?

Once you get these out here, you know, what are people really going to find interesting? And so once you get these feedback loops between, you know, what the market is really responding to, I think that the set of applications that get a lot of attention, I mean, I have my guesses. I have some things that I think are going to be really compelling in VR. But, you know, at the end of the day, most people I know in this space have their own set of guesses and we have fun, you know, debating, you know, what direction it's going to go. but it's a really big open question. I think even before we get to that point,

we have to ask, like, how is, like, like, our thesis is that VR is not just going to be for gaming. It's going to be, you know, it's like the next platform. I think this is what we believe, the Oculus as well. It's that this kind of, like, VR is an enabling technology, which makes you feel like you can be anywhere. So that doesn't mean, that, I mean, obviously that's great for entertainment. You can think, you know, wow, I'm in this, like, super cool,

gaming world so much fun but like there's like way broader applications beyond that um it's just a question of like figuring out how and when that will happen so what does it mean for me be able for me to be able to move into any space at will and especially when those spaces can be populated by either people I know and love and want to interact with even if they're not around or people that I've just met so it shouldn't be forgotten that oculus just released their they're so social alpha, right, which is the ability to go in and watch a movie or video in a fake movie theater with other people who can be from anywhere around the world. And it's very early and the level of interaction is very crude, but the fact that they thought

this was, you know, a very early thing to push out there, you know, very quickly says something real, which is that, you know, I think that for a long time, you know, one of the knocks against VR from various quarters has been that it's going to be this very isolating experience. there has long been a thread, I think going, if you go back to, you know, Jaron Lanier, you know, back in the day and then certainly continuing to today of people saying, you know, wait a minute, really what's interesting about VR or one of the most important things that's interesting about VR is that it allows us to share experiences with other people. The technology is going to take a little while for it to catch up, but, you know,

I'm one believer and I know that there are others who think that, you know, social is not just something that's a nice to have in VR, but for many application domains, a core and fundamental capability that is going to be built into everything. And we don't even really even know how powerful it's going to be, but it's absolutely essential. And I think a lot of people, when we talk about capturing experiences and sharing them, think about using 360-degree video or in the future light fields to capture what's happening around you in terms of big events or concerts. And I think that that misses kind of how important in our lives the boring times are. I think that we're going to see a lot of social usage of people getting in their VR headsets

and quickly scanning in the room they're in and then hanging out with their mom for 20 minutes instead of doing the phone call. Playing poker, I think, is a great example of people of using. There are some people who I interact with you say, look, the day that I can sit around a poker table with a few friends and have the kind of eye contact and social experience that a good poker game has, that's when I know this technology will have arrived. On the one hand, it's very mundane. On the other hand, when you break down what's necessary to have a good poker experience, it's pretty non-trivial. There are a lot of things you have to be able to get right that just aren't available today. One thing that was really popular on, like, I think there's this VR company called AltSpace.

One thing that they found was really popular was people hanging out and watching YouTube videos together. And it's actually really interesting when you realize that the average person, I think, over 40 spends like seven hours a day watching TV or something like that. imagine if they spent that time, you know, hanging out with virtual friends instead. I think that's just kind of a nicer experience for everyone. One of the most compelling experiences that I've had in VR was when Next VR at the very beginning of the basketball season live streamed the first Warriors game. This is before, of course, we knew the Warriors were going to go on and be the awesome team that they are this season.

But so this was great. And the camera view is right at the half court line. You're on the court and you are kind of like if you were live at a tennis match, you know, I mean, your head is going from side to side as you're watching this game. You're that close. And basketball is a phenomenal game for this because the ball is big enough that it works with the current resolution. And they just, in a lot of ways, totally nailed it. And I was hoping it was going to be awesome.

And, you know, I have been pretty unimpressed with a lot of things I've seen in VR. This was the future. And I want to be able to pay money to do this on a regular basis. But I was at that game alone, right? And there's no way for me now. I mean, my wife was, you know, sitting next to me on the couch. She had no idea what I was doing and it was this, you know, on some level, incredibly compelling, but also a very isolating thing. So I think a huge, you know, nut to crack is how can I share those entertainment experiences with those, with people who might be in the same room with me or might be across the country?

And I think once you've got that, there's just a really compelling combination. So one additional point about presence. So some people have said that even if you're with like a very crude model of another person, It's even because if you have VR presence, it kind of still feels like you're with people. I think anyone who's tried the Oculus Toy Box demo can probably relate to that in that seeing a floating head and hands with no emotions whatsoever, no textures to convey gender or race or anything like that, you still feel like you're in the room with that person, no matter what the distance is between you. And just seeing how people gesticulate while they're making a point and knowing that like, oh yeah, this is. It feels so completely removed from the experience of like a Skype call where, sure, you might even see the actual person doing that, but it doesn't feel like they're there.

So there's an interesting concept from neuroscience called filling in, which has been around a neuroscience for a long time. And now I see a lot of people in the VR community using it because it's very useful, which is that once – and it's a little complicated in the sense that it's hard to predict sort of a priori or from first principles exactly when the brain is going to sort of kick in and start filling things in. but, you know, one example that people will often cite in VR is that if you get a certain amount of, for example, visual cues right, then even very incomplete or crude haptic cues will feel very complete because your brain is expecting them and it's going to basically fill in all those details that may not actually be there in the real stimulus. So, you know, the most important thing is don't have your cues be disconsinent. Don't have them contradict one another. So make sure you don't do that. The next level is you could have those haptic cues missing completely, but then if you can go one level beyond that and just have that mere suggestion of haptics or, you know, whether it's, you know, the audio or whatever, you can get a lot of bang for your buck with even relatively, you know, low fidelity and low resolution cues.

The flip side, though, is that it's not always obvious, as I was just saying, what counts as low fidelity. And so there's a lot of experimentation that's taking place to figure out sort of what is the minimum. sort of additional cues that we can provide. But once you hit that minimum, you can have these incredibly rich multi-sensory experiences that really kick you into a new level of presence. Toybox is one great example of that, and there are a number of others that people are discovering. As one who thinks about kind of the public perception of VR, though, I will say, though, that it's a little disconcerting that people are already, you know,

people see a computer strapped to your face and they go, oh, that's what makes people think it's isolating. I like the term face toy. Yeah, that's my personal favorite. But over time, you know, as you add things like haptic feedback and, you know, making it feel like your arm actually faces resistance as you do something in an experience, you know, are we all going to be strafing on like full on body suits so that we can feel when our arm is nicked in a game? Oh, I bumped into that wall. Are we going to go full ready player one, right? Where they're sitting in a room by themselves, yeah, in these full body suits, probably catheterized and, you know, not leaving for quite some time. Yeah, I think there are big questions here about what is going to be socially acceptable and what's going to be socially acceptable in the home. What's going to be, you know, when you get into AR, you know, what about when you're out in the real world?

Obviously, there was a whole set of issues around, you know, acceptability of Google Glass, which is, I think, the example that most people have in their mind when they think about it. You know, I don't know. I think one of the advantages of VR is that because it's fully immersive and you're not expected to interact with other people face to face, and you're doing it to a large degree in the privacy of your own home or, you know, or who knows, it kind of gets a pass on that at the beginning, but these issues are going to be very, very important going forward. And it's really hard to predict how social acceptance is going to evolve, right? Yeah. Some people are saying that the whole full body suit thing

might be like a, you know, a throw, a comeback for the arcade. So maybe not everyone will, you know, buy the camera kit and body suit and eye tracker and every single input device, but you might have some places which have all of those things. And then your average user might just use mobile VR. Or it's an exceptionally standout experience where you have, you feel presence unlike anything you could do at your own home for, you know, tens of, that. Each setup would be tens of thousands of dollars and things like that. In a very meaningful way, virtual reality works better when it's tricking your brain along

its various senses. You know, you're seeing things that look realistic, you're hearing things that make you feel like you're there, maybe even you feel something that makes you feel like you touch something in that virtual space. So I'm going to, using that framework, I'm going to pivot to an interesting application set that I think is going to be very important for virtual reality, which is education. You know, we talk about kids learn, some are better visual learners, some are better audio learner, some are better when they write something down or experience the actual task of doing that. So with virtual reality, kind of making your brain see these things at a very, like, intimate working with your wiring level.

What do you think that means for the future of education? You know, Google is talking about shipping cardboard headsets to schools. But if, you know, people are actually able to experience something where they're walking around seeing the Constitution be signed for the first time or, you know, things like that, what does that mean? Yeah, well, the obvious first application for VR in education is things like, you know, flight simulators, learning to drive, things which have like a very visceral, like, physical hand-eye coordination component.

So think of it, like, in the Matrix when Keanu Reeves, like, puts on the head sense, like, oh, I know Kung Fu. I mean, maybe that's kind of what it will be like for a VR. When it comes to more abstract things, it can also really be really helpful. Because, you know, if you have the more intense and varied your memory, like, that helps with remembering facts. And, you know, like, seeing something on screen, you know, fixing your mind more than reading it on the book, seeing in VR even more so.

So I think there are a few things. And as I have a three-year-old daughter, I've been thinking a lot about, you know, how she's going to experience school and, you know, if and when she'll ever even go to college and all these things. And so I think that for VR and education, I think that it's fairly obvious that both history and science education should just be in VR. Like, why on earth would I want to, you know, read about a, you know, these important historical events rather than be present within them.

I mean, so that to me is just a matter of producing the experiences. I would love to have that job. Yeah, absolutely, absolutely. And there are a couple of studios that are doing this. So the one experience that my three-year-old daughter has watched more than any other is this Apollo 11 experience, which, if you haven't seen it, super awesome. They've done great assets for the Saturn 5 rocket and then inside the capsule, and she's actually been on the moon and the whole thing.

it's not even in full release yet, but I have like the pre-release version. It's great. So for that kind of thing, like, duh, go do it. Like, let's get this out there. I mean, that's really obvious. Going back to what Taku was saying, where I've personally been pretty interested is how do we translate abstract concepts into VR. So for example, you know, CS concepts. So I don't really get distributed systems. It's never anything I really studied, you know, intensely. I've never had to implement them. But I know that they're really important. So what does it mean? for a lot of different, you know, machines and computers to have to reach consensus. You know, what, how are those messages getting passed around? I haven't seen the VR version of the thing that teaches me that, but I bet when I do see it, I'm going to understand those concepts much better than if I read about them and maybe even better than if I implemented them. You know, maybe, maybe not. I'm not sure.

But I think that I'd really like to see those kinds of abstract concepts be rendered, you know, in these environments, and I want to sort of see what that means. I don't know if you guys have heard of Brett Vicks' Kill Math Project, and so his idea was to take upshot concepts like math and translate it into just symbols which you can kind of, which kind of translate the ideas, but in a kind of visual and indirect way. Like one thing you might do with VR is to teach chemistry

would be like have a virtual chemistry set. You can just move things around and experiment with them, and then you have, you know, you don't have to get one yourself, you can do dangerous things. And you can see things like in the traditional gaming space. You have things like Curbel Space Program, which is kind of an animated, silly take on building a rocket and trying to send it on missions to the moon

or wherever else. And a lot of it is just trying to build it. Oh, no, I failed. Let me tweak this one thing. Oh, I failed again. And you do that a hundred times. Then finally, you understand some of the basic concepts of physics and how these things actually work. And you look, and NASA is actually now trying to make a kid's version of this game. That's maybe a little bit more simplified, but still gets the same fun aspects, you know, still captures that. And there's a few areas, you know, Bo, you brought up science education, like looking at cells splitting, which do students engage with more? A piece of paper, you know, printed on an 8x11

sheet where you see cells split in eight different frames or an animation that shows it in distinct colors or in a 3D model that shows what's happening inside the cell at a very detailed That's right. And so high fidelity, one of the Philip Rosedale's company, is using cell biology in particular as, I think, their sort of initial application for exactly this reason. And I haven't been, I haven't seen it in VR, but I've seen it projected on a screen, but it's awesome. I mean, it's great. And I think that that kind of topic is just, is perfect for it. One of the use case I hear talked about a lot is a VR dev environment. Nearly every developer I talk to about VR wishes that they could have, a headset that they could have, a headset that they could

put on and have, you know, 10 or 15 screens to code on. I know, Bo, you've had... Right, and one of the complaints right now is that people who are developing for VR, particularly, are taking on and putting off, or... They have to take their HMD on and off hundreds of times throughout the day. So just at a minimum, you want to be able to create VR within VR. And then beyond that space, right, there's the whole tool set. So there's creative tools, there's the equivalent of the Adobe Creative Suite.

there are development environments of other types, you know, what will an office environment within VR look like? How will that work? That's clearly a big area. We know we need a lot of innovation there going back to the design and interaction issues we were talking about before. So a ton of interesting stuff to be done there. But then beyond that, you know, sort of the tool space, which has to happen or else, you know, we're not going to get anywhere. There are a number of others which we can at least touch on briefly. So one that I'm personally very excited about is, you know, roughly speaking, how do you kill video conferencing, right? So, you know, everybody, I don't know if you guys have an investment in any video conferencing, but I'll just go ahead and say it. Okay, then this is my opinion only. Everyone hates video conferencing. I mean, it's just, it has a lot of real, real drawbacks. And, you know, you can't really make eye contact with people. You don't really know who's paying attention.

It's really, you know, it's really difficult to have a natural and free-flowing conversation. just, you know, a lot of real, real issues there. So if you could give people the ability to, you know, be in a room together and have a productive business conversation together, there's a ton of value there. And I think a pre-existing market. And I think that at the firm, we definitely see instances of that, not quite implemented through VR, but we use beam robots, where, you know, it's the head on a tablet driven around by wheels.

And you can actually, yeah, it's, you're engaged with this person. It feels like you're talking directly to them because it's. It's just their head looking back at you. You can tell when someone's distracted. Like, no, we're actually engaging eye to eye. And when you see that, it's so much better than having to do a insert video conferencing application here. It just feels like it's more productive because they're there with you. Even if there's no delay whatsoever with a video chat, for some reason, it feels like there's this distance.

Right. And it's important to point out that within business communication, there are probably at least dozens of individual use cases. Some will be better with the telepresence robot. Some are probably really well served right now. Others, you know, others much less so. I was really excited to see that AltSpace, and of course their early adopters, they do their stand-up meetings in VR right now. And I think that's awesome. I mean, it's obviously weird and awkward to a certain degree right now, but that's definitely the future.

So, you know, then there's business communication, but then you can also imagine, you know, personal communication. So I think, you know, FaceTime has been wonderful for grandparents everywhere. And so the question is, you know, what is the equivalent that we can provide in VR? And, you know, my personal opinion is that that's probably a little bit further away because, you know, you're dealing with normals, roughly speaking. And so how do you get that, you know, how do you get that level of adoption? But I think there's also, you know, a big, big opportunity there. And there are issues just, you know, what kind of avatars will people have? Do you just go with the untextured blank face of toy box? Do you try to scan people's faces in? Do you like, do you like? let them create their own avatar as like a video game, what works, what's broadly acceptable? Yeah, absolutely. I think that the choice of avatar, both resolution and, you know, realism is incredibly interesting. I can make cases for any of the possibilities that you just laid out, and I feel like I have basically no idea which way it's going to go, because this gets into the social

acceptance issues, and they're really dynamic. Yeah, I think one interesting other point is that if you have really, really good telepresents, that affects, like, how we behave in other ways. Like, think about how many miles people travel every year for business, you know, how far people go to visit their friends, you know, driving, like all those kinds of things. I mean. Yeah, and that starts to point out that VR, like pretty much any very general purpose technology and computing platform, is going to have some really interesting second and third order effects. And it's, we can sit here and make guesses. And at the end of the day, a lot of it's going to come down to some pretty fine-grained issues of economics. and the quality that's provided and a lot of things.

And so, yeah, I think absolutely, you know, one of the possibilities is that self-driving cars may be less important than we think they're going to be because people find it less important to travel for the same reasons that they do right now. I'm not saying that's going to happen. I'm saying it's entirely possible and that VR at least, you know, plausibly has the power,

you know, to have an effect on that level. And, you know, could, you know, just as self-driving cars are probably going to have a pretty big effect on, you know, how real estate value is distributed because certain areas will be easier and harder to get to, I think, you know, VR too. I mean, if we have to travel less, if commuting daily for a job becomes less important because, you know, VR telepresence, you know, really delivers on its promise. Then in 10 or 15 years, you know, what effect is that going to have on, you know,

where people choose to live relative to where they work and, you know, in all those issues? I think, you know, there are a lot of big possibilities there at this point. we're just guessing about exactly how it's all going to net out. Just so that we kind of cover the spectrum of what the nerds really appreciate in our VR-enabled future and what people in the real world are going to want to experience, people who aren't making it up as they go in their garage, I think an interesting aspect of VR and in kind of related category, augmented reality, is fashion. You know, the idea that once you have this avatar that represents you in all of these spaces,

could you use that avatar to try on virtual goods that then correlate to the purpose? purchases you'll make in the real world. I think shopping's interesting because e-commerce has been growing steadily over the last 10 or 15 years, but it's still only like 7 to 10% of all the things that we buy. And given that, you know, people are willing to like instig up their groceries, they're willing to buy pretty much everything online and clothes. So, like, the things that we buy the second most often, and yet it still quite hasn't made the leap to e-commerce yet.

And part of the reason with that is, you know, you don't want to buy something which doesn't fit and then you have like you know some kind of wonky thing that you have to return later um and so i think vr is interesting in that way it's it's worth noting that um like online retailers have been trying this problem for a long time they've been they have these like virtual models and they're like oh maybe people can try things on this way but none of them have really caught on so it might be vr that gives it the final push right so if you look at the argument for the continuing relevance of you know brick and mortar retail you know 10 15 20 years from now it's that there are certain physical experiences that you need to have in the context of that store

that you're not going to be able to get at home. And so, you know, the idea behind VR and also AR, which we'll get to in a second, is that it really eliminates that, that final reason that you actually would ever have to go to a store. So this is, you know, far more, the argument is far easier to make in some areas. In other words, fashion is, is a great one. Another one that I I saw pointed out recently online is, you know, furniture and sort of, you know, large goods. And this sort of gets to the whole area that I think is really interesting of, you know, architecture and interior design in general, which is that if I can, you know, envision a space and put new things in it and try them out, holy crap, do I want to do that, right?

That is so obviously, at least, you know, at a conceptual level, better than the alternatives. As someone who recently, you know, bought and furnished a house, oh my gosh, would I have preferred, you know, A, when I was shopping for houses to be able to do virtual walkthroughs, and that's, you know, it seems to me like a slam dunk case that, you know, will be out there very soon and there are companies that are going after that opportunity. But then once I had that house, you know, being able to virtually furnish it before before actually getting these very, I mean, I still have a set of closet doors sitting in my garage that are probably never going to leave my garage because they didn't turn out to be the right ones. And I, you know, just couldn't know that based on what I saw. online. If I could have hung those doors virtually, I wouldn't have those doors sitting in my garage for the next 10 years. And that's, you know, that's pretty compelling. And with something as intimate as fashion, you know, a men's small, medium, large, woman, zero, six, and 12, those all mean very different things at different retailers. And, but at the same time, they all have the absolute data of how much fabric is in every different size and every different garment. So why not in

the future put your exact measurements into a virtual reality viewing application and see exactly how that piece of clothing drapes over your body size. I mean, what, that seems like, you know, we talk about Amazon having two-day shipping or even same-day shipping and, oh, that lets you return things easier and you'll try more things. If you can know exactly how this shirt or dress will look on you. I shop for a lot of clothes online and I basically accept that only about half of them are going to fit.

And right now, for me, it's still so much more convenient as me to do it that way. I just sort of, I eat that, but that's clearly, you know, suboptimal. There's a lot of room for improvement along exactly those lines. It's worth noting that fashion is a very low-tech industry in general. I know if you guys have heard of Zara and the whole fast fashion movement. That was applying a very small amount of technology, you know, just like a little bit of analytics, a little bit of prediction. And then they just became really huge just from, you know, measuring what people were buying. And so, you know, I think, like, you know, the fashion industry could really get a huge boost from something like this.

If you used the DK2 and gone on Oculus share, or if you know, now have the new gear VR and go, you know, check out the experiences section of the Oculus store. One of the things that was pretty quick to arrive on those platforms was, oh, here, live on a fantasy beach for 10 minutes. And I think that that kind of, that's a very early example of something that also could be big in the future, which is the idea of, you know, virtual tourism or virtual travel, the idea of if you can simulate presence and being somewhere and capture environments in near photorealistic detail, why not just go there? in a headset versus making the real trip and spending maybe thousands of dollars to go see

something where, you know, frankly, you can't actually go up to that piece of artwork and touch it and feel it. So what's the difference? Yeah, actually, the company, John Fierre, they started it because one of the founders went on a trip to, like, Monument Valley in Utah. And he thought, oh, my God, this is so amazing. I want to look at it at any time I want. And so after that, they made this 360 camera recording rig. So what it does is it's like a ring of cameras in a row. and it records video in 360 degrees. So then you can go, go to some really cool location,

record what it's like, and then replay it back to yourself on your headset. The other way people are doing capture is through doing light field tech photography. So that means it records like how the light is falling everywhere. So it's another camera rig, but it kind of reconstruct the scene in a slightly different way to 360 video. It includes lots more information about how the lighting falls

on all the objects in the scene. And so that's just another way of playing back the experience of being in some cool place that you can visit. For listeners who maybe haven't tried kind of the scope of these experiences from 360 video to light fields, I think it's important maybe to differentiate

in practical terms what those technologies provide in terms of ease of capturing content and then what the experience will actually be like for the viewer. So with 360-degree video, it's actually using traditional camera sensors and optics and then stitching together that footage so that it's 360 degrees around you and maybe there's even a 3D effect so there's some depth to what you're looking at.

With Whitefields, on the other hand, you're actually capturing this as like 3D points. So when you're viewing it, it's not just something that's a video playing at a fixed distance from your head, but there's even the possibility of, you know, as we talked about earlier with the DK2 and having camera and being able to look in and what that means, with these lightfields, you can actually do that and see what you captured from different angles. And, you know, again, it contributes to the feeling of being there. But at the same time, there are a couple of downside. Saku, could you kind of detail maybe what the downsides are to light fields, given they sound so appealing? Yeah, lightfield's the main problem is to just say it's a long time to capture.

You know, to take a scene and then take everything in it and everything about the lighting information and then, you know, replay it back. Like, that's a lot of computational work. But on the plus side, that's a very parallelizable problem, so we might see more of it in the future. So, and one way of thinking about that is when Lytro, when they recently announced their light field capture system, yeah, they announced a camera,

but alongside it as part of the system is the giant rack of basically disk and compute that you need just to, you know, capture and then preview the content that you're capturing. It's so it's just, you know, at a computational level, it's a massive problem. The amount of data that you're capturing on a, you know, bits per second basis is just truly, truly massive. It's worth mentioning that most VR isn't done using either 360 or light field. It's done in the same ways games are made using 3D modeling and rendering, like the same techniques people use to do special effects in movies. Yeah, no, and in the medium slash long term, these also, it seems like there's potential films to kind of blend together.

Right. you're capturing light fields, but then you do some rough computation on it to build, you know, a underlying 3D model, and then you slap the light field on top of it as a texture, essentially. Right. And then you have a light field that you can also maybe interact with in a game-like way. And it's fascinating to think about what that would enable then, where you capture Rome, but, oh, also you can go into the shop or, like, open the door, or, you know, in maybe something less, you know, light touch, you know, something like a video game like Grand Theft Auto, where you use. use lightfields to capture a real city and then you play around with it. Right. Like one cool thing about Unreal is that they've been making this big push to include

like Hollywood quality assets in their game making. So like a lot of VR developers have been using like these assets which are being captured from like motion capture studios or like, you know, sometimes even light field stuff. They incorporate it into their games and so you have this kind of blend between the traditional video game way of doing things and then the new light field. field or motion capture way of doing things. One more thing on lightfield. So light fields are really hard to wrap your mind around.

And a podcast might be the worst possible medium to explain light fields. So I just want to put in a plug and maybe we'll put it in the show notes for a great set of talks that came out of the USCICT lab that explain it. And they're awesome. And if you want to understand this topic, it's far. explained by the absolute best people in the world to be doing this explanation. And that is what you should watch in order to sort of get a handle on the topic. So I'm really incredibly excited about the possibilities for travel and VR.

I think it's worth pointing out that this is an area where the quality of the experience matters a ton. And so I think there's the great Paul Saffo line, which is never mistake a clear view for a short path or something along those lines. So clearly being able to transport yourself to somewhere that you can't actually physically visit or that would be too expensive is just a huge, and I strongly believe in this application. But for it to be more than a novelty, and as someone who's tried these out, I think that whether it's light fields or even more exotic additions like redirected walking in a room scale environment, so really quickly redirected walking is where. you give someone the impression that they are in this sort of infinitely large environment by subtly changing the way in which the virtual scene is rendered so that they are basically prevented from, you know, walking into a wall, but they don't feel like it. You walk a little bit to the left, but what you think you're going in a straight line.

Yeah, that's right. And there are all kinds of tricks where, you know, as people are either, you know, going through doors or turning by 60 degrees, you can turn them by maybe 80 degrees and they won't notice and it's super cool. So, you know, there's going to be quite a bit that's necessary for me to really feel like I'm at the pyramids of Giza to the level where I don't actually need to go to the pyramids of Giza for this to have an experience that I would consider to be comparable. But once you've gotten to that point, I think there's just an incredible amount that is unlocked. I think that one of the great potentials for VR is just to give far more people in the world. the ability to have the awesome experiences that currently only a relatively very small number of people can afford or are in the right place at the right time to experience. So, you know, being courtside at a basketball game is one kind of prosaic example of that.

I mean, you're basically removing the scarcity of court side seats. You are transitioning that scarcity from the world of Adams and, you know, it's now, you know, zero marginal cost, world of bits, right? It's sort of that standard transition, but applied to what up until now were very physical embodied experiences. And I don't want to claim that this is going to be a quick transition or that people are going to consider this to be, you know, a fair trade anytime soon. But I do think that is, to my mind, sort of one of the most exciting sort of long-term trends that we have here. I'm actually surprised. We managed to cover a lot of bases in VR. We talked about the technology and the medium itself, but also kind of design.

paradigms and things to consider. So I really appreciate you guys, you know, joining and sitting here for an hour plus to really go and dig into all of these aspects. Thanks for having us. Yeah, it's been great. Thanks a lot.