CppCast - VR Development

Episode Date: September 16, 2015

Rob and Jason are joined by Nicolas Lazaraff to discuss the current state of VR development with C++. Nick is a VR/AR engineer who is passionate about bridging the interface between computers and humans. Currently he's VP of Software Development at OTOY focusing on VR and AR ("mixed/digital reality"). He was a cofounder of everyAir, a pioneering P2P game streaming application which was later acquired. Before that he worked at Microsoft on Office 2010 and 2013. News Stack, Heap, Pool Dependency Injection in C++ using Variadic Templates Nicolas Lazareff nzff.net Links Balls Away - Game on iTunes App Store CppCon: C++ for cross-platform VR development OTOY & Oculus: Render the Metaverse Oculus Connect Keynote: Future of VR Panel Down the VR rabbit hole: Fixing judder

This episode of CppCast is sponsored by JetBrains, maker of excellent C++ developer tools including CLion, ReSharper for C++, and AppCode. Start your free evaluation today at jetbrains.com slash cppcast dash cpp. And by CppCon, the annual week-long face-to-face gathering for the entire C++ community. The 2015 program is now online. Get your tickets today. Episode 27 of CppCast with guest Nicholas Lazarif recorded September 16, 2015. In this episode, we discuss memory allocation techniques. Then we'll interview Nicholas Lazarus from Atoy. Nicholas will talk to us about the current state of virtual reality development with C++.

C++ developers by C++ developers. I'm your host, Rob Irving, joined by my co-host, Jason Turner. Jason, how are you doing tonight? All right, Rob, how are you doing? Good. Are you all prepared for your talk next week? Sure. You don't sound too confident there jay i'm prepared for my my main talk but then uh the chai script open content session that i uh signed up for that one i you know trying to figure out exactly what i want to share and how flexible i want to make it for

to be able to show whatever people ask about. Right. Well, you kind of know Kaiskrip like the back of your hand, I'm guessing, so you can do whatever you want there, right? Yeah, that's true, but I still want to have some structure to it, not just say, hey, what do you guys want to see? Sure. And you just got the CPP cash shirts in the mail, right? I did, yes.

They're looking good. Awesome. So look out for Jasonason at the conference i'm not sure what your plans are for handing them out but uh i do not yet have any plans we will have 20 shirts available and uh i guess whoever's most interested in getting one needs to find me first yeah okay at the top of every episode i like to read a piece of feedback. This week, I actually have a comment on Reddit from last week's episode with Scott Myers. Someone was commenting on the discussion about auto with braced initialization resulting in initializer list as opposed to an integer or whatever type you might be expecting. And someone pointed out a reason why you might want this to happen where you could have a for loop

and iterate over your initializer list of integers. And yeah, I guess that is a valid use case, but as someone else points out, maybe we could just make a special rule for for loops? Yeah, it looks like it was Scott himself who pointed that out. Oh, it was Scott. You're right. I'm sorry. Yeah. Why not just make a special rule for for loops if this

is the only, like, real use case for that? Makes sense to me. Anyway, we'd love to hear your thoughts about the show, so you can always email us at feedback at cppcast.com, follow us on Twitter at twitter.com slash cppcast, or like us on Facebook. And you can also review us on iTunes, which we really appreciate, as it helps us get more listeners to the show. So joining us tonight is Nicholas Lazarif.

Nick is a virtual reality and augmented reality engineer who is passionate about bridging the interface between computers and humans. Currently, he's VP of software development at Otoy, focusing on VR and AR. He was a co-founder of EveryAir, a pioneering peer-to-peer game streaming application, which was later acquired. And before that, he worked at Microsoft on Office 2010 and 2013. Nick, welcome to the show. Hi. Well, thanks for having me. That's quite the change from Office into VR and game development. Yeah, it is.

I guess I've always really been interested in graphics, and one of the kind of main reasons that I ended up, you know, leaving Microsoft and working on Every ended up, you know, leaving Microsoft and working on Every Air was, you know, just kind of general passion for games and rendering graphics. So what Every Air was was something that let you stream from your PC or Mac to any iOS device. So, you know, you could play World of Warcraft on your iPad or iPhone. And actually, there's some pretty impressive videos on YouTube

of people that mastered the on-iPad controls to play sophisticated games like Assassin's Creed. So I guess, you know, from Office to VR and AR is a big leap or, you know for different industries, but it's something that I've always been really interested in. Is it something that you were kind of doing as a side project while you were still working at Microsoft? Game development?

I wouldn't say game development was a side project while I was there, but games in general, just like I think a lot of programmers, the reason that you get into it is because you play video games around 10 or 11 or even younger and you're just you know you become enthralled by them and you decide that you want to make them too um so yeah i've been making games um for a long time uh you know writing 3d engines, and just generally toying around with game ideas. Every year it came along just basically as a test project right after the iPad came out, and then one thing led to another. It really was, in some sense, kind of spurred by,

wouldn't it be great to be able to play games on the go with your iPad or your iPhone? And obviously, games that aren't mobile games or companies that probably never do or never will port their games to mobile. Obviously, World of Warcraft is one of those. Right. Just out of curiosity, when was Every Air? When did that first start? So we launched it in 2010, and then by mid-2011, we got acquired.

So it's been a little while since it came out and was bought. That sounds pretty fast to get acquired, yeah. Yeah, we were doing pretty well, and the company that acquired us really had a kind of similar long-term vision, and we ended up working on something very similar, actually, but not games-related. So Autodesk has a product called Autodesk Remote, which lets you stream. The theme is the same, obviously,

stream from a computer to a mobile device. In this case, obviously, it's focused on Autodesk applications. So basically, we built the core technology for that app. Interesting. All right. So since we're talking about your background right now, I'm curious what your first game project was since you said you've been working on game engines for a while. Personal or otherwise. Yeah. So you can find a game that I worked on on the App Store now. A friend and I, actually the same guy from Every Air, he's an engineer at Google.

Well, anyway, long story short, you can check out the game on the App Store. It's called Balls Away. We wrote a 3D engine basically piece by piece over time and eventually decided to make a game with it. The game isn't doing spectacularly well, but it's something you can download and check out. It's kind of a Fruit Ninja-ish maybe spinoff. Okay. 3D models and some pretty cool animations. I think it's pretty fun.

So anyway, you can check it out. Alright, cool. I'll put a link to that in the show notes. So I had a couple quick news items to run through before we get into the interview with Nick. The first one is this blog post I found called

Stack Heap and Pool from Bulldozer's blog. And I believe we've talked about some of his posts before. I believe so. And this one is just analyzing performance between stack, heap, and object pool allocations.

And, you know, part of the post is kind of expected. You know, obviously stack allocations do much better than heap allocations, but it's kind of interesting to see some of the numbers, especially seeing how fast the pool construction and allocations took. So he shows here, I guess he allocated like 1,000 objects that were 1 megabyte in size, I believe. Sounds right.

Yeah. thousand objects that were one megabyte in size i believe sounds right yeah and in release mode the heap allocations took a total of 371 milliseconds stack took 35 milliseconds and then the pool was constructed in 307 milliseconds and the actual allocations took zero milliseconds um nicholas i know that the pool pool type of allocation is probably more popular with game developers. Is that right? Yeah, I think for resource management. So actually one thing that may be worth calling out in terms of the pools is their use in real-time, whether audio or video streaming um so you know part of the way that the

application that i worked on in the past and even to some extent the streaming stuff that i do at work today relies on ring buffers and yeah you have to basically allocate a decent chunk of memory up front as you get these encoded frames, and then you need to decode them and then eventually push them off to the GPU to get displayed. So I would say that, you know, both memory kind of pools and stack allocation have their uses. I guess, you I guess these days, dynamic heap allocation is sort of frowned on,

but I think even that has its uses. Very cool. Is there anything you wanted to add to that, Jason? No, not really. Just some interesting stuff to think about. And I guess, as usual for this kind of topic, the Reddit conversation itself is something to read if you really want to get some more info. People discuss the various merits of

different types of pool. Right. And the next article I have, also we pulled from the C++ Reddit, and this one was on dependency injection and C plus plus using variadic templates. I'm not going to go into this one too deep because it's a pretty code heavy blog post, but, uh, it was pretty interesting to see, uh, the use of variadic templates as a way of doing dependency injection. Um, now Jason, I know we were talking a little bit before the show and, the two of us, I guess, haven't done much with dependency injection, right? I definitely haven't.

The, the, the concept is a little lost on me. I'm trying hard to find where I would have used it in a project. How about you, Nick? Have you, uh, done much with dependency injection pattern? Um, not really. I think the classic, uh, argument for it is for testing purposes. So if you have one object that inside of its constructor creates other objects, that makes it difficult and or impossible to unit test.

So generally you have some other factory that creates the objects for the object that you're trying to test. But I can't say that just in general the code bases that I've worked on have really used templates heavily. Yeah, I definitely understand, I guess, the use case for it, but at least in C++ I don't think I've really ever applied this pattern. So if I ever want to, though, this definitely looks like a good article for reference. So Nick, let's start talking to you about VR development. And I might as well say up front that, you know, I don't think Jason and

I know much about virtual reality development at all. So you're really gonna have to fill us in on what the state of, you know, the SDKs and everything is around VR. Where do you think we should be starting? So it almost a little bit depends on kind of what you are most interested in. I'd say there's kind of a large split at the moment. You can choose to go the desktop route, in which case there's the DK2, which is really easy to get. It's just a couple hundred dollars. If you're kind of lucky enough to convince Valve to send you the Vive, that's also another route to go.

That would be obviously in the desktop for a small amount of invested money. Personally, the other option and the one that I'm excited about is the mobile side. So you basically take an Android phone. In the case of what Google has, there's a cardboard. And in the case of Samsung and Oculus together, there's the Gear VR. It is a little more expensive, especially in the case of the Gear VR, because you need a Note 4 or later, the later Note 4s.

So it's basically about a $1,000 investment just to start to play around. But what's really compelling about mobile is that it's really easy to get up and running, and it's untethered. So... Maybe we should go over that a little bit more um the dk2 is that the oculus rift device yeah so um it's for if you aren't familiar it's dk for development kit and it's two because there's been one two of them um oculus also has the uh gear vr which is a project that they're working on with Samsung.

And in that case, well, like I was saying, you plug in effectively a high-end Samsung phone into the headset, and that serves as the CPU and the display. Oh, so it is an actual separate headset. It's not like the Google Cardboard box kind of thing. Yeah, exactly. So the cardboard is, you know, It's not like the Google cardboard box kind of thing. Yeah, exactly. So the cardboard is, you know, it's actually quite literally just a piece of cardboard. And there are people that are making them, taking kind of the cardboard idea and making them out of plastic or, you know, other materials. The Gear VR, it's a full-on device. It costs $200. uh the gear vr uh it's a full-on device it costs 200 bucks um and you also need a very specific

basically set of phones that are only the high-end samsung ones just curious about the google cardboard because i haven't had a chance to try that one out is it really as immersive as wearing something like the oculus rift um so i mean the cardboard is probably not the best example okay uh it's effectively just using the um you know the sensors on the phone the gear vr is augmented by uh sensors uh on the head mounted display itself it also has higher quality optics it has a little track pad on the side and in addition oculus Oculus and Samsung have worked closely to make changes to Android itself to improve VR experiences. Now, all that said, one thing that all these mobile devices lack is what's called, you know, the jargon for it is quote-unquote position tracking. So if you think about your head, there's kind of two large elements to it.

One is the direction that you're looking at, the orientation of your head itself. And then the other one is the position of your head in space. So none of the mobile devices give you CV1 and the Valve device called Vive or HTC Vive, they are significantly better VR experiences. There's no question. Okay. So what languages are you going to be writing in if you want to create a game or some other type of application for these devices? So on Android, the realities of Java are inescapable.

And to some extent, Cardboard really kind of wants you to use Java as much as possible. I think Google in general wants people to use Java as much as possible on Android. That said, you can, there's nothing stopping you from calling C++ code from Java. Now, on the other hand, there are some Java components to the Gear VR Oculus Mobile SDK, but it's set up in such a way that it really encourages you to write primarily C++. Okay, so you have to use some Java, but you can call into C++ the same way you might call into C++ for a regular Android mobile app. Is that right? Yeah. So maybe one way to say it is in the case of Cardboard, you won't be able to avoid any of the

hairiness of the JNI. But in the case of Gear VR and Oculus Mobile, you actually can avoid the JNI because Oculus, the SDK, has handled most of that for you. Okay, so if you're writing an Oculus app, you can write it only in C++ if you want to? Is that right? Effectively, yeah. I mean, you're not going to be able to avoid having some Java layer inside of your code,

but you won't have to write any of that. Oculus has written that interface for you. Obviously, the asterisk there is if you want to interface somehow with the Android OS, well, then you might have to make calls from C++ up through into Java or kind of do those calls the other way around, Java into C++. So Oculus makes both an Java and C++. So Oculus makes both an SDK and the hardware, and you don't necessarily have to use their hardware with their SDK?

No. So they make the hardware. In the case of the desktop, they make the Rift. In the case of mobile, they make the SDK, and Samsung makes the hardware. Oh, okay. Yeah. But also their SDK is, at least for the moment, tied to their hardware. All right, so I'm curious from just a practical 3D programming perspective.

You are making a virtual reality game, and do you have to specify that you have two cameras in your scene and they are spaced so far apart? Or do you just say, this is my camera and I want you to display it on a VR display and it does the right thing? You can do it either way you want. The simplest way is the SDK will effectively actually on both desktop and on mobile, it creates some version of what you call like an app object or a program object. And that program object gets callbacks that let you draw either per eye if you want to or per frame. So I would say the most common way to do it is you just render what you want to render per eye. And in terms of setting up of cameras, you let the SDK in some sense do that for you.

Okay. Okay. So we're talking about all these different devices, and I'm assuming that a lot of developers who are making games are going to want to target all of these devices from their application. Is there any support for being able to, you know, kind of create a cross-platform VR app currently? So I would say that probably most people that, if not all people, that are currently making games in VR use either Unity or Unreal. And in that case, you're shuffling the work of cross-platform, which is a significant amount of work to the developers at Unity and Unreal. So

you know, in terms of tools beyond Unity and Unreal, there really are none. If you want to write something that's... In particular, if you want to write something that's not a game, you end up writing stuff yourself

and trying to be as cross-platform as possible. So they have built-in support for these VR headsets? Yeah. Okay. I think Unity is kind of really investing heavily in good game tools for VR. And Unreal is also, but I think at the moment, certainly Unity has more developer mindshare. And they make it actually quite easy to develop and to make games for VR. Okay, so we've mentioned Unity before on the show,

and I believe their engine is written completely in C++, but most developers writing a game in Unity are actually using C Sharp. I'm not too familiar with how Unreal works. If you're making an Unreal game, are you using a scripting language or are you writing in C++? I think you, well, it probably depends on the size

of the company that's making the app. Okay. You can write Unreal games in C++ and you can also write them in their high-level scripting language. They have their own scripting language, or do they use something else? I think it's called... I'm not super familiar. I think it's called UnrealScript, and if I had to guess, it's some offshoot of JavaScript.

I'm not sure if they have support for other languages. I think I've heard of UnrealScript, but but yeah i don't know what it's like either i think it's more it might i mean i'm not without 100 being sure um just based on um you know chats with uh other developers working in vr typically if you uh are making something in unreal you're likely to you'd be C++. I want to wrap this discussion for just a minute to bring you a word from our sponsor, JetBrains. C++ is on a comeback lately, isn't it? This language has been around for so long, yet it's hard to find a good development tool for it. Luckily, our good friends at JetBrains have just released a new IDE for cross-platform development in C++,

and we thought you should know about it. This IDE is called CLion, and it's compatible with Linux, OSX, and Windows. It relies on the well-known CMake build system and offers lots of goodies and smartness that can make your life a lot easier. CLion natively supports C and C++, including the C++11 standard, libc++, and Boost. But that's not all. Editing CMake files gets a lot easier with CLion, thanks to completion for commands and file names and automatic updating of CMake files. With its in-depth understanding of C and C++ code, CLion brings you smart, relevant code completion. It also gives you full coding

assistance, like customizable coding styles, key maps, and project views. You can instantly navigate to a simple declaration or usages too. And whenever you use CLion's code refactorings, you can be sure your changes are applied safely throughout the whole codebase. When it comes to version control systems, the IDE supports Subversion, Git, GitHub, Mercurial, CVS, Perforce via a plugin, and TFS, with a unified interface for all of those. It even has a built-in terminal where you can run any command without leaving the IDE, locally or remotely, using the SSH protocol. A powerful language needs powerful developer tools. With CLion, now you've got those. Download the trial version at jetbrains.com slash cppcast dash clion. And if you are a student or are developing an open source project,

use CLion for free, courtesy of JetBrains. Well, I was curious, if we can continue with a different question here. What we haven't mentioned yet, I don't believe, you are going to be speaking at CBPCon on this topic, correct? Yeah, I'm going to be speaking on exactly what we're talking about here, which is using C++ for cross-platform VR. In this case, it wouldn't be using Unity or Unreal. This is right from scratch.

Sort of the theme, or maybe not the theme, but the storyline of the talk is an evolution from writing code entirely for each one of the applications to later relying almost entirely on C++. And then finally actually going to some kind of mix between entirely C++ and some scripting layer that would allow people that are artists and designers to work concurrently along with engineers. So this is what you do in your professional development? You use C++ directly from scratch? That's right, yeah. I would say that the code that I work on is probably 99% C++.

Okay. So do you find that that is putting you at a disadvantage compared to the people who are using these other SDKs and gaming toolkits? Not really. So I think Unity and Unreal are great applications for game developers and I think it's effectively at this stage really almost impossible

to provide anything better than what they have. If there are drawbacks to Unity and Unreal, it's that, well, A, they're big, beastly kind of applications. They're not super easy to get into. But also, B, they're really focused on game development, and that actually leaves out a non-trivial percent

of the application development or software development that's happening today in VR and AR. A lot of these apps that are being made aren't games. They're 360 video apps or photo browsers or chat apps or, you know, things like that. Interesting. So a good example, maybe, I don't know if it's the best example, is an app called Tilt Brush. That app was made, I think, in Unity or Unreal. Actually, I think it was made in Unity, and it's a painting application.

Is Unity the best tool to be making painting apps? I'm not sure. But certainly it takes a greater level of sophistication to use Unity than perhaps these lower-hanging kind of fruit apps require, like a photo browser or a video browser, something like that. So what type of scripting language are you using along with C++? So that's a good question. So at work, the language of choice is Lua. And I think two potentially interesting candidates, one that John Carmack has proposed is Scheme.

And the other one is just JavaScript because, you know, people are just super familiar with it. And if we want to be kind of expanding and making it easier to develop VR apps. One important way to kind of approach people is in a language that they're familiar with, and JavaScript obviously is really widespread. That is true. JavaScript is making its way everywhere, yes. Yeah, if you didn't know what the laughing was,

it's because I have my own embedded C++ scripting language, Nick. Yeah, maybe that's another alternative scripting c++ uh one of the nice things i mean it almost goes without saying about a scripting language um it's not just that you can potentially kind of have uh a separation between the people that are working on the lower level performance, more kind of C++ side of things, and designers and artists. Hello? Sorry? Yes. I kind of had a thought to finish there. Oh. It's okay.

Oh, sorry. I thought I was cutting off. No So I don't know if you want to edit this. So what I was going to say was that the other nice thing about a scripting language beyond kind of the separation between kind of quote unquote engineer and quote unquote designer is that it lets you iterate quickly. You might be able to live on the device see things change or animate. I think that's certainly one of the things that has drawn Carmack to propose what he's calling VR script. I think in a couple days, actually, at Oculus Connect, one of his sessions is going to be called Live Coding in VR. And just from what I gather um um from his twitter uh part of that will be showing something where like you know he'll be typing a command like show a cube or something and boom it'll appear in automatically in vr so you can kind of quickly see your ui or your app evolve as opposed to having to

recompile and redeploy it over and over again. So I think there are some kind of non-trivial advantages to scripting as well. Right. That reminds me of this video I saw a couple months ago where someone, I guess it was with Oculus Rift, they had a code editor within Oculus Rift and they were typing in it and using it to create objects that they would then see you know also in the virtual reality environment i'll look for that video it's really interesting maybe you've seen yeah yeah i've seen that that's a really cool video so one of the things also that uh you've seen that video is um the person is demoing uh editing of a shader

inside of v. Right. And it's just kind of really pleasing to see the effect of the shader happen live as you're sitting there and editing. So yeah, that's a great video. I'll have to look for that link for the show notes if anyone wants to watch it. That sounds interesting. I haven't seen it.

So can you tell us about some of the VR projects that you've worked on? Yeah. So the one that's kind of launched and you can find on the Oculus mobile app store is called Orbex Media Player. And probably the thing that has picked up the most buzz with that is this Render the Metaverse contest that the company that I'm at, it's called Otoy, sponsored with Oculus. And, you know, for those that haven't seen it, you can obviously find it on the web. But the idea was we want to get these 3D artists to create really high quality and well thought out and designed VR scenes. And then once per month, award them, you know, award the best people. And I think there's been some really just fantastic results that have come out of the contest.

Very cool. That is interesting. What's the name of the contest again? The contest is called Render the Metaverse. Render the Metaverse. Okay. Yeah. The contest is called Render the Metaverse. Render the Metaverse, okay. So what is your personal favorite application or use of virtual reality that you've seen?

I would say the thing that has drawn me the most of all the things that I've seen is when you have an object that's in the real world and, um, you somehow have representation of that also in VR space. So this might be an interesting video to also put in the link is, um, you have two people wearing these, uh, the valve headset, the Vive. And, um um the vibe also comes with effectively a wand it looks like a wand that's the uh you know the input mechanism and what you can do is you toss the wand um to the other person in real life obviously you're you know you're holding it and you're throwing it to the person and the other person can like they'll see it in in their virtual space they'll see the wand fly to them

and they'll be able to catch it. Those types of things where you have this interesting mix of real and virtual, I think they're incredibly compelling. And then when you see people do that, it's not just that they take off the VR goggles and they're like, whoa, I felt like I was on a spaceship or something. It's like these beaming smiles of, holy crap, this was – I don't know why throwing a piece of plastic to you was so cool.

But it was the coolest thing I've done. So I think in general, at least to me, these types of mixed virtual and real experiences are super cool. So since we're talking about mixed virtual and real, in your bio you mentioned that you're a virtual reality and augmented reality engineer. Maybe we should go into what the difference is between those two for someone who might not be familiar. Yeah, so it's possible that in the long term there kind of is no difference between the two and that's why people are starting to coin various different words like mixed reality or digital reality

but certainly in the short term or where we are right now is virtual what are called virtual reality experiences they are head mounted displays that encompass your full field of view. So an example of that would be the Oculus, whether it's on mobile or on the

desktop. You can't see the real world through those goggles. Augmented reality is more like, you know, the glasses that you see, like Google Glass, and more recently, Microsoft announced the HoloLens. In that case, you're still wearing goggles, but you see through into the real world, and in the real world appear things, you know, whether it be, you know, a movie or, you know, some 3D object, or, you know, whatever. Right. Yeah, the HoloLens videos really impressed me. Yeah, the HoloLens is, you know, super cool. What's interesting, you know, and why I say that it's possible that at some point they just become one thing is, if you think of what happens, say you're wearing something like the HoloLens,

and then there's a button that says, okay, show me the real world. Presumably, if you have some kind of visor in front of your eyes, you could also toggle some button that says, okay, make me entirely virtual. Just black out the real world and give me some virtual representation. So right now, the only two kind of AR devices you'd say would be the HoloLens, which isn't out yet, and Google Glass, which, correct me if I'm wrong, but Google Glass kind of got tanked. I don't think people are really developing for that anymore. Yeah, I think the project there at Google had a tremendous promise. And then for one reason or another, they've kind of chosen to shutter it and then i think very recently maybe in the last year or so reopen it and reinvest heavily in a different

idea that said it's not just um the hololens and google glass that have uh augmented reality goggles or glasses there's actually quite a few companies that are also making these. Epson has a pair of glasses. There's a company called ODG that also has a pair of AR glasses. Oh, okay. And I would say that they are Google Glass-like. Maybe the HoloLens is the most sophisticated of them all, but the experience is different, say, comparing the Google Glass to these ODG glasses or the Epson glasses. Right.

So what do you think are some of the current limitations keeping VR and AR back from mainstream adoption? Well, I think they probably have their own, the split between the two, they have their own unique set of challenges. Putting aside AR for a moment, because at the moment the glasses are bulky and people aren't, it's not a fashionable thing. I think one of the reasons that Google Glass failed is people were just, they just creeped out by

it, right? They would see potentially someone recording them all the time and it just looked kind of lame. The challenges in VR at the moment out by it right like they would see potentially someone recording them all the time and you know it just looked kind of lame um the challenges in vr at the moment um you know there's a quote somewhere i think by carmack was like the first challenge of vr is just not making people sick so a lot of people put these things on, and whether it's by not entirely understanding the right set of experiences that VR is best for, combined with just general lack of hardware and software that's good enough. So one of the main problems of VR at the moment is we would all like to have higher res displays um there's a at one of the oculus keynotes um mike abrash which is uh kind of chief scientist

at oculus said that kind of the ideal resolution would be 16k by 16k per eye which is that's already way more than any display that we have today and And actually, in addition, so he used to work at Valve, and he has a great series of blog posts on the Valve blog talking about latency in VR and AR. In one of those, he mentions that 1,000 FPS is kind of the dream frame rate. So we're already super far away from that, just computers being able to power 1,000 frames a second, but also just displays. A super expensive display today is maybe 120 frames a second.

So just one overall way to say it is there are a lot of performance issues in both hardware and software, and there's a long way to go to get to the ideal state. At 1,000 frames per second, he really thinks it would be necessary? Yeah. At 16K. So, I mean, if you're going to put some links in the footer here. I can send them to you. You can also find them.

It's on the Valve blog. He has several posts on latency and frame rate and judder and VR. They're just very, very good. I can't recommend reading those enough. Wow. That's way better than the 720p TV that I've got. Right, exactly. So there's obviously a cost.

There's one consideration, which is how do we even get a display that, forget a computer powering it for a second, how do we get a display that can just run at 1,000 hertz? Yeah, and we're not talking about a large display. We're talking about a tiny little display that's right in front of your eye. Yeah, exactly. And not only does it have to run 1,000 hertz, talking about a tiny little display that's right in front of your eye yeah exactly yeah so and it not only does it have to run a thousand hertz um it also has to be 16k by 16k per eye and you know

most of us have two eyes so you're talking about just a ton of a ton of pixels wow so yeah i mean of course that's the ideal um between what we're at now and some ideal future, which may not come in any of our lifetimes. There still can be tremendous progress, and there's still plenty of a zone that's just good enough. It's good enough to have compelling experiences. And I think this time around, the displays, some of them are 60, some of them are 90, some of them are 75 um and actually sony's is as high as 120 frames a second um that's good enough um and you know i think shortly in five maybe 10 years the uh frame rates on the displays will continue to go up and uh so will the resolution but then also kind of our our expertise in creating kind of high quality compelling content for vr will also go up

so do you think there might be like an uncanny valley for vr looming between here and there where like things look really good but our brains still know that they're not right you know like the current problems we have trying to rendering render humans um i i think it's almost like there there is a difference um in so when you talk about uncanny valley you it's it's much worse uh when it comes to humans and right you know if you kind of extend that uh that chain of thought to vr there are some VR experiences that are going to be very difficult to do in the short term. That said, there are some VR experiences that won't be that hard to do or certainly improving at a high rate in the short term. So I don't think there will be, you know, I don't think that some uncanny valley applies broadly across all vr experiences

um and uh you know as a result there will be experiences that are good enough and all that said actually you might think that um your brain is really difficult to fool but it's actually surprisingly easy to get you to feel things like vertigo or just to lose, you know, your sense of, of, of where you are in the real world.

And even actually just like for your brain to just turn off on, on, you know, on the real world and an easy way that you can see that with people that are new to VR and actually even people that are experienced with it is you just have them turn around, like getting your, you know, swivel around in the chair maybe three, four times. Or in the case of some of these mobile devices, you can just stand up and just spin around. And you'll very quickly, your brain like loses its kind of sense of where you are in the real world. And, you know, you believe that you're there. You truly believe that, you know, you're in a dinosaur And it, you know, you, you believe that you're there, you truly believe

that, you know, you're in a dinosaur cave, or, you know, whatever. Well, that's pretty interesting. Well, is there anything else that you want to talk about? Maybe going back to your talk before we let you go? Any other kind of insights about, you know know what you can do as a c++ vr developer to share more code um yeah i would say uh well you obviously check out the talk once it's posted on youtube um which i'm presumably it will since cppcon tends to uh to post them um but in general i think one of the reasons that c++ is having kind of you know a second golden age or a renaissance is it really is a great language for kind of targeting multiple platforms and in addition to that its performance strengths are you know particularly suited for obviously

game development it is the the language of probably most game developers but um you know it's doubly suited for vr which is an incredibly um you know performance kind of uh demanding uh environment and curiously enough um with a game for example um you might care about performance because people get annoyed by latency or by skipping frames. But in the case of VR, something like latency can actually make someone puke. They can get physically sick. So in some odd way, the stakes are much higher in VR. And, um, you know, that adds kind of an extra, uh, kind of asterisk to why you should use C++ and why you should be conscious, uh, performance conscious. Well, in this case,

it's because you actually might make your users like physically ill. Um, so do you have to keep any of those airline bags around your office? I've actually never seen, I've, I've only heard of someone um puke you know on internet or on forums and stuff uh but it's very easy to uh to have an experience where your stomach feels it right away so even people that are like super kind of um rugged against vr motion sickness if you put them in a game like tf2 and then you just have them play with WASD and a mouse, the fact that you're moving the camera so fast with the mouse and also kind of moving in space

will almost for sure give you a stomach ache. Huh. Interesting. Yeah. There's something about kind of rapid motion that your stomach just doesn't agree with. Well, you're what, experiencing the motion, but your inner ear's not experiencing it, and your body's not feeling the change in inertia, that kind of thing, I guess?

Yeah. I think people have proposed kind of various, I'm not sure how scientific or accurate they are but one of the stories that i've heard you know many people kind of uh tell over and over again maybe this is just like a lore thing i don't know if it's actually true but what they say is um since your brain knows that you know in real in the real world you're standing still um but in this virtual space uh you're moving around you know at light speed uh it somehow thinks you know it kicks into some reptilian brain system that thinks that you're hallucinating um and you clearly must have eaten some poison berry or whatever so it's time

to eject that now of course you know the asterisk on that is i don't know if any that's true but that's something that people repeat a lot that's kind of part of uh maybe an old wives tale or something of vr that's interesting yeah okay well nick where can people find you online if they want to follow some of this or uh just learn more about what you're talking about um so i think you can check out my upcoming CppCon talk. Actually, another now that reminds me, if you want to check out another really interesting panel talking about the future of VR with John Carmack, Mike Abrash, and I think Palmer Luckey, maybe one or two other people. It's on YouTube if you just search for like future VR Oculus Connect. It was a panel from last year.

Super, super interesting. You know, hearing these guys talk with one another and actually sometimes even disagree about what the future is or what it does and doesn't entail. And I guess in terms of finding me, I just recently kind of started writing a little bit on a blog. You can find it nzff.net. There's only one post, but if you want to read it, it's on Lightfields. And send me feedback. I'd love to hear it. Okay.

Well, thank you so much for your time, Nick. Yeah, thanks for having me. Yeah, thanks for having me. Thanks so much for listening as we chat about C++. I'd love to hear what you think of the podcast. Please let me know if we're discussing the stuff you're interested in, or if you have a suggestion for a topic, I'd love to hear that also. You can email all your thoughts to feedback at cppcast.com.

I'd also appreciate it if you can follow CppCast on Twitter and like CppCast on Facebook. And, of course, you can find all that info and the show notes on the podcast website at cppcast.com. Theme music for this episode is provided by podcastthemes.com.