Microsoft Research Podcast - 091 - Inside AR and VR, a technical tour of the reality spectrum with Dr. Eyal Ofek

Episode Date: September 25, 2019

Dr. Eyal Ofek is a senior researcher at Microsoft Research and his work deals mainly with, well, reality. Augmented and virtual reality, to be precise. A serial entrepreneur before he came to MSR, Dr. Ofek knows a lot about the “long nose of innovation” and what it takes to bring a revolutionary new technology to a world that’s ready for it. On today’s podcast, Dr. Ofek talks about the unique challenges and opportunities of augmented and virtual reality from both a technical and social perspective; tells us why he believes AR and VR have the potential to be truly revolutionary, particularly for people with disabilities; explains why, while we’re doing pretty well in the virtual worlds of sight and sound, our sense of virtual touch remains a bit more elusive; and reveals how, if he and his colleagues are wildly successful, it won’t be that long before we’re living in a whole new world of extension, expansion, enhancement and equality.https://www.microsoft.com/research

I see an opportunity to generate reality which is not limited to the physical law that reality abides to. Nowadays, all our life, we're in the cage of our body. We see things from our point of view. We can go wherever our body can get us and what our senses can do. And suddenly, we could see things from someone else's point of view. We could extend our senses. And as much as it's life-changing for me, it could be even more life-changing for people that currently have limitations.

In a non-physical world, then there's less importance of what my physical ability is. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Huizinga. Dr. Eyal Ofec is a senior researcher at Microsoft Research, and his work deals mainly with, well, reality. Augmented and virtual reality, to be precise. A serial entrepreneur before he came to MSR, Dr. Ofec knows a lot about the long nose of innovation and what it takes to bring a revolutionary new technology to a world that's ready for it.

On today's podcast, Dr. Ofak talks about the unique challenges and opportunities of augmented and virtual reality from both a technical and social perspective, tells us why he believes AR and VR have the potential to be truly revolutionary, particularly for people with disabilities, explains why while we're doing pretty well in the virtual worlds of sight and sound, our sense of virtual touch remains a bit more elusive, and reveals how, if he and his colleagues are wildly successful, it won't be that long before we're living in a whole new world of extension, expansion, enhancement, and equality.

That and much more on this episode of the Microsoft Research Podcast. Al Ofeck, welcome to the podcast. Hi, great to be here. You're a senior researcher at Microsoft Research, and your work is centered around reality, particularly as we've talked about augmented reality and virtual reality, or AR and VR, as I will refer to them for the rest of the podcast. And even this idea of mixed reality, which we didn't talk too much about, before we get started, let's operationalize those terms. Obviously, there's some overlap. How would you define the different kinds of realities that you're working on from a technical perspective? And how are they the same or different from each other? And going to augmented reality or virtual, it's a different levels of intervening with those senses. Let's say that right now I'm looking from my eye and I see the reality.

But if I will, in part of my field of view right now, put something new there, then this is what people call augmented reality, where you see reality, but there's something, I don't know, surreal that appears there. If I continue adding more and more content there, I see less and less of reality. And eventually when all my field of view is filled up with new content, this is what we call virtual reality. But those terms, again, are just point onone continuum. And the same can go for audio and other senses. So if you're not experiencing anything additional, then it's real reality. Yes. As far as we can define that. Exactly. Exactly.

That's a whole other podcast. Let's talk about grand aspirations for a second. As we're getting started here, what is the big idea behind AR and VR? What are you trying to accomplish? What gets you up in the morning? First, philosophically, we understand the world around us through the senses. It's a whole new world. We could see how digital, not physical things could change our reality. This is a whole new concept. Up till now, we know when we use, say, digital programs, we use them in a device. We take out the phone, we look at the screen,

and we see something there. Changing reality around us, that sounds a bit weird, right? Yeah. But it opens so much possibilities. It's a bit different than what people maybe experience right now. But I see that as an opportunity because this could change reality in a way that will make it better for me. I could be a superhuman in there. I could understand what's happening not next to me. With a flick of a finger, I can be from this meeting to my workstation.

And there's opportunities that are beyond what we know in our physical world. We can reach things that we cannot do. We can enable things that we cannot do. Even for people that have limitation in a physical world, it might be an opportunity to level the playing field. Interesting. We looked at ways that it could affect conversation between people, because you suddenly have a lot of information all the time with you, and it doesn't have that social effect of getting the phone out while talking and looking.

And suddenly you can do a lot of, for example, we found out that we can help introverts to have better conversations. Great application for the software field. Well, I want to come back to that and come back to it heavy a little bit later in the podcast. But even with what you've talked about right now, 25 years ago, I did not have the smartphone that I could find out anything I wanted to, anytime I wanted to, anywhere I wanted to. And so in a sense, I have a superpower already. It just happens to be in my pocket, not in my head right now.

You're right in a sense that when we got the internet, we immediately sort of upgrade our ability to know information. The augmented virtual space here is about getting that information without any visible device when you need it. We're looking at a different way that it can change life of people, whether it is working, obviously, and how we can make things more efficient this way, to how we can play, to how we can communicate and interface. Because one thing that you can see today is that all those screens around us basically reduce communication between people. You can see a family sitting around the table, and each one is looking at its own phone. What if the information was around them?

So at least their line of sight will include other people. And what if the system can show you private information, but it will incorporate that other person in a way that you might still keep being immersed, but unable to communicate with other people. You got your start in computer vision, and you were a bit of a serial entrepreneur before you got your PhD. Yes. Tell us about some of the projects you worked on back in the day, and then talk about the importance of timing, especially as it pertains to the little failures that eventually lead to big successes. Yeah, I had several startups even before doing my PhD, and it's fun.

You could do things which are totally new and be able to show that to the world and enjoy the impact that you see around it. Like we did a drawing application, which includes also 3D manipulation way before Photoshop. Or we did a game engine that includes a simulation of ray tracing. And that was in 94. My enjoyment was when we did things which were really new and pushing the envelope. And that was maybe different than what's needed from the business point of view. At some point, I decided, okay, let's go all the way for novelty. And I joined Microsoft Research.

I want you to drill in on this idea of timing. Because when we talked before, you mentioned that you were drill in on this idea of timing, because when we talked before, you mentioned that you were kind of on point, but too early in the game. And even you mentioned a Street View camera prior to Google's. So that actually was after I joined Microsoft. But you're right. I mean, if I look at my startups, they could have been much more successful if they were a few years later. It's true for many things that you try to push in. And you have to understand that the fact that you have a prototype working might still have a long way to get into the product. So, for example, we actually did a project where we mapped two cities,

San Francisco and Seattle. And again, this one actually wasn't that early, but it still was a year and a half before the first commercial one. So just speculating here, what do you think then contributes to the world not being ready for your great idea? Well, it's not that the world is not ready. But basically in general, I know you talked with Bill Baxton in the past, and he has this term of the long nose of innovation. Right. Where we see different new technologies that came out.

And many times you see 20 years before they actually reach maturity. Yeah. You framed AR and VR as a revolution. If we look at this through the lens of other revolutions, why do you think AR and VR qualify as revolutionary? Okay. Let me explain that, and I can come out with different directions. To show what's the difference between what they enable or what I expect them to do and what we had before. It's not about, oh, I can put on a headset and now I'm immersed in a game which is all around me. No, that's not the revolution. I

mean, it's nice and pretty cool. Yeah. But there's several things that I think are a major difference, which actually most of them don't yet see in the application in VR today, or AR for that matter. The first one coming from technology is suppose I right now want to write an AR VR application. How do I do that? Right now, what people do is very much like they did in, say, writing a game for Xbox. I design a game, I develop the game, test it, clear all the bugs, and so on. And now, once I have it, I ship it as a product. And everybody has to do exactly what I did. Basically, making sure that whatever is the platform that they use to run it

is exactly as the platform that they use to build it. So it's standardized. Yes. And you see that on every software. Now, what I see as an AR VR is the fact that the software came out of the computer and now live in our environment. So it's opened up sort of a Pandora box. How do I write such a software? because it should run differently on different people. So we set that as a goal and we started working on that almost 10 years ago and trying to see how can we make sure that software will be on one hand similar for everybody, the same playability, the same story, the same task that

you need to do. And on the other hand, be dependent on how is the room around you? Are there other people? Your particular environment. What time of day it is, and so on. That's a huge technical challenge. It is.

So that's a technical side of it. Yes. And it's dependent on all the sensors that are mapping my environment. And so it's personalized for me. So it has basically two elements. One is an element of understanding what's the world. So understanding what's out there, what are dangers, what are opportunities that we can use, that's one thing. And on the other hand, understanding what is the applications that we try to do. Let's say I want to tell a story.

So a story will have these three chapters that I need to visit. So I need to first visit chapter one, chapter two, and then chapter three. I need to put them somewhere in your environment. And so this analysis of the environment on one side and this breakup of the narrative that I need to give you on the other side, and then matching between there. All right. So this is a complete shift from how people have written software and what inputs they've used to write the software. That's right. The use of VR and AR has been described as sporadic and episodic. Yes.

Why is that the case right now and what will it take, do you think, in terms of progress in these technologies to move us from sporadic and episodic to a more pervasive adoption? Yes. Eventually, what we want is, I want that if I'm coming in the morning, I will be able to control my workspace regardless of how it looks like, whether I'm sitting in a bus or that I have a small office with no windows. I want to be able to do whatever I want. I'm not a touch typist. I want, as I type on Word, that I will see my fingers just beneath the line as I type and not where they really are on the table. I want to be able to reach and fetch something

which is really far from me. I want to right now jump to a conversation we had last week, and suddenly I'm in that meeting room, and the whiteboard has the writing as we left it last week. And again, I didn't drive to do that. I want to visit someone in a different site and be able to walk with him in the site, but I'm still in my building. You know, those people that walk around talking to the air because they have a Bluetooth earpiece. Think of someone that walk around the pavement,

but now he's also wearing glasses and he's talking and he actually is walking right now in a different continent. All these are possible. However, if you look at what's happening today, there's several limitations. One is hardware limitation. The headsets are still not that good, heavy, limited on battery power, connected to big computers. And tracking up till a few years ago was limited to a room. And nowadays they have inside-out cameras that enables you basically to walk around. But even more fundamental is the applications.

So this has to fundamentally be changed to be more flexible. And again, that's the direction that we try to push. When we talked about the revolutionary nature of AR and VR, you talked about the technological challenges. Right. And I know that there's another angle to this,

which is the social challenges. Right. So talk about that, unpack what the challenge is there and what we need to be thinking of here. Okay. So let me first say I mostly see things as an opportunity, right? I see an opportunity to generate reality, which is not limited to the physical law that reality abides to. Nowadays, all our life, we're in the cage of our body. We see things from

our point of view. We can go wherever our body can get us and what our senses can do. And suddenly, we could see things from someone else's point of view. We could extend our senses. Let's say right now I'm in a car and there's a traffic jam. What if my point of view could jump from driver to driver up until the point of trouble and I see what's happening there? What if I could reach wherever I want without any effort? And as much as it's life-changing for me, it could be even more life-changing for people that currently have limitations. In a non-physical world, then there's less importance of what my physical ability is. And it could be a leveling playing field for people.

Not just physical, as I said, it could also help people just by guiding them in the moment to do things. So there's a lot of potentials. For example, we did the work in the past that was trying to see if augmentation in augmented reality can help people generate better connections. Suppose I'm sitting in the train and there's someone next to me and we start talking and we have our means of finding out if there's any connection. Do you know this? Where did you learn?

Where did you live? And so on. And sometimes we end up the conversation and nothing happened. And sometimes we just fall on some commonality between us and this conversation starts flowing really fast. And wouldn't it be nice if I could start the conversation this way? So we did an experiment like that.

We gave people that never met, and we asked them, what do you want people to know about you? What are things that you want to talk about? What are things that you don't want to talk about? And then when they met, we try to enhance the conversation and we want to see several things we want to see if people will be annoyed by the fact that they are getting things while they need to talk right and would the other person be annoyed by the fact that I'm reading something while he's talking and again

we want to see if this will enable the conversation. Will it flow better? Would you look for subjects? Would you use what we did? What did you find? So interesting enough, first, we're very blind for what the other person is doing. And we are very bad at discovering if the other person has a whisper in his ear of things to do. Second, it depends on people. There's people that are great talkers, and they don't need any help. And those people, A, didn't use our system, and B, thought that this was sort of, what am I, in the kindergarten?

Why are you guiding me? Right, right, right. While other people that are not that fortunate actually used it and it helped them. So what I'm hearing is individual preference on how you like to interact with people, what you want people to know, and where you like to focus your reality. That's right. When we talked about the science of doing AR and VR, you mentioned three areas that are important to get it right.

We've talked a little bit about it, but I want you to drill in on what they are and why they're super important. Okay. So the first one, I don't know if I need even to dive too deep in that, is safety. Obviously, right now, we are changing how people perceive reality, but at the end of the day, they live in reality and reality can come back and bite them. Literally. Yes. So they need to be aware if there is any object that might collide or might fall. And sometimes they're not aware even if they don't have any device.

So first, you want to make people safe. And second, now you have a story that you need to give. And I'm saying story. It could be a game. It could be an application just as well. And you want that this information that you give about the world will not break that story. So, for example, if I'm in a spaceship and say my mother comes in with a plate with cookies to give me while I'm playing in VR, then she may appear as a floating robot that comes in and say there's nourishment and cookies will appear somewhere on a desk in the spaceship. Will it be her voice or will it be a robot voice? That's a good question, right? It could be a different voice,

but at least when the voice is associated with something you see, you tend to connect that. So that's that sort of mixed reality where mom's actually bringing cookies, but it's a robot. Right. So we want to keep the experience untouched by the fact that the whole world is trying to interfere with it. Okay.

So that's sort of safety and mapping reality to augmented reality or virtual reality. And what's the third one? A third one is privacy. We want to be able to give a lot of options, but on the other hand, keep all information at your place. Ale, if I let my imagination run, I can see how AR and VR could, in essence, democratize our lives by extending, expanding, and equalizing everyone's experience with reality. You know, sitting in coach class and maybe thinking it's first class, hey, I have more virtual leg room even if my legs are still blocked. And also that your older car could seem like a fancy car.

Tell us some other areas where the concept of resources and abilities might be equalized by virtual and augmented reality becoming a part of our lives. Okay. Suddenly we got freedom. We disconnected ourselves from reality. So what can we gain as a result? On the other hand, we don't want to totally disconnect reality. We want to be a contributing citizen and be able to work and interact with people and so on. So what we say is what if we could filter out the reality? So things which are boring could

become more interesting or enlightening and so on. We have a paper right now called Dreamwalker. I walk every day from my home to work, and it's the same path, basically. And instead of doing that path, what if I could, say, walk in Manhattan? Well, I do that. So again, safety first, but then we can actually generate a new environment around us. Or maybe I'm doing a routine job. I need to bring a parcel to some office. And here we can transfer you to maybe a fantasy world where I'm running up with a chalice up the hill fighting dragons and getting it to the castle. I mean, there's different opportunities. That's not the direction that currently we're working. We're working more on getting people to be more in reality, but still having more content, but it's a possibility.

How does this play out in the ability world? I know you have some research and some papers in this area. Yes. So one disability that we're trying to look at is maybe the furthest away that you would associate with virtual augmented reality. that we're trying to look at is maybe the furthest away that you would associate with virtual augmented reality. And that is a problem with vision. Because people associate virtual reality with such a visual language. What if a blind person use it? So there's several reasons

for that. Let's talk first about low vision. Low vision is some limitation with your vision that cannot be fixed with glasses. There could be people with a lot of atmosphere, a lot of small details and maybe darkened environments and so on. And we wanted to make that reachable for people with low vision. So we can use the fact that this is a world that we know a lot about. So we can apply filters that will make things look clearer to people, mark where objects begin and end, mark what could be graspable. And when we try that with a set of people

that had low vision problems, we can see how it's improved their ability to do that. Now, if you go even further for people that are totally blind, yes, they cannot see the display, but they can hear what's around them. We can even sense things such as virtual sensing of a cane that hits an object that does not exist there. And as a result, they know that there is an object and they navigate a virtual environment. Right, right, right. When you talk about the increasing quality of our virtual senses, mostly we're talking about vision and hearing, but touch has remained more elusive in this arena. So tell us why, from a technical perspective, touch has been so difficult to simulate. And then talk about the progress you're making in haptics and some of the hot papers and projects you're working on. So first, how did I got to touch? Because,

as I say, we started with augmented reality and the fact that applications will look different in different environments because we actually see the environment. Now with virtual reality, I just put myself into a virtual world. I no longer see my environment around here. What do I care about the fact that your environment is different than someone else's environment? And the answer to that is mostly touch. I can walk straight forward and in your environment, I will be able to do five steps. In someone else's environment, after two steps, I hit a wall. So I need to take that into account. And this means both how to avoid things I don't want to touch in the environment as well as how can I touch things which are not

there, right? So if I right now go into current application, I see very nice graphics of virtual worlds around me, and I hear 3D audio around, I can hear the birds coming from the right, and then I reach my hand to touch the bird, and nothing there. So we wanted to give the sense of touch. Now, this is hard. And you can see people working on things such as exoskeleton. We tie ourselves into some robotic suit. It's very expensive. It never gets out of the lab.

And what we wanted was something that we can see coming to consumers. So we said, you know what? Let's model the objects that we actually are going to hold. So instead of exoskeleton, we went through controllers. Because right now we also use controller for Xbox and people that use VR right now have two controllers to both track the hands as well as some buttons to press. What if those controllers could change their shape and be a representative for objects that we grab? And we had a whole set of different devices that enables you to grab objects of different sizes, feel that they're rigid or maybe compliant. I could press them and they will inflate back

when I release the pressure. Maybe I can connect the two controllers in a way that one time they can be rigid as if I hold a box in my hand and I cannot bring the two hands closer. And then within a second, the two hands are free and can move as they want. So we have a whole research direction

that try to see what can we do more and more in a way that will be simple and cheap and robust so people can put that into products and on the other hand, very realistic as much as possible for the users. We believe in people using it in real environments. So real environment has objects. For example, if I need to touch a virtual wall, maybe there's some wall around that I can actually

touch. The problem, that wall is not where my virtual world is. So if I can fool you, since you don't see your real hands, we could bring your hand, your real hand, to the real world while you see your virtual hand touching the virtual wall and that timed experience will convince you that the virtual world is real. And we did several works on that, which we called haptic retargeting, which tries to, as much as possible, use what's around you as a mean for haptic sensations. Of all the research going on at MSR, AR and VR may provide the most Hollywood-ready script for a dystopian sci-fi movie, if I may say.

So, as we've just said, the goal of your technology is to fool me with digital trompe l'oeil, as it were, and empower me to believe I'm superhuman. Yes. Somewhere on the spectrum of reality, virtual reality, augmented reality, someplace in there. While I maintain a grasp of real reality, because that's important for me not to kill myself or injure myself, what could possibly go wrong? So despite the fact that you say you want me to believe, but you don't want me to believe too much. Right. Is there anything about this that keeps you up at night? So there's several things that we're thinking of, and I must say that most of them are not unique to AR and VR in this sense. First, how do I trust what the system gives me, right? How do I trust what I read on social network? How do I trust what I read in a paper?

In some sense, AR, VR is easier because they deal with the world just around me. So taking off the headset will reveal me the real world no matter how much I try to fool you. So we're very limited on what we can actually convince you. On the other hand, I will add the opposite problem. We don't want you to believe us too much. What I mean by that is we totally trust the technology, right? We shouldn't. Technology, even if all our input is correct,

it might not be a full input, and there are noises, and there's things that it doesn't know. So if I, for example, have a machine that tells me, here's a straight line, and put the nails along the straight line, I should always have in the back of my mind,

maybe it's not a straight line. And that's another thing that we try to do. So people will not be just totally passive, totally trusting the system. How are you doing that? Several ways that you can do that. One is you could involve the person in the decision making. Look around, tell me, and things like that. And also use a language like,

I think you should put the nail there. What do you say? Even with the language, you can use things that will generate some kind of inner retrospective, throwing the liability on you. So you will wake up and say, oh, now I need to really understand what's happening. You know, something that you said earlier, I just wanted to bring up because I was just in Florida and we wanted to go see some alligators. And the gators are mating and moving around in March,

April, May, and not in the middle of the summer because guess why? It's too hot. Right. So they're all underwater. So we went out for this ride and saw, you know, the nose of one alligator. What if I had a virtual reality set of glasses and I could go gator watching in the wrong season? I have the thought of what if I, say, go to Pompeii with my wife and I put on a headset, which sounds ridiculous. I'm already there. Why do I need to put a headset, right? Or if I put a headset, why did I even bother coming to Pompeii?

Why did you buy the airfare? But the thing is, it's the feeling that I have right now of the air flowing, the smell of the food, the people around me. But the people around me, instead of being tourists, will be wearing togas. And the cars around will be chariots. And the buildings will be full. And my wife will be there next to me. And whenever I want, I switch to a full reality. I can totally see that because having been to Pompeii recently,

it would be, I'm looking at the ruins and what if the world were built in front of me as it was, and then I could go back and forth, even experience the volcano eruption. Maybe not that. All right, let's move on. You said we should do a series. I think you're right. You're an interesting guy, Ale, in a lot of ways. Tell us your story. What got you into a life of high-tech invention? And how did you ultimately end up in a life of research at MSR? I would say I started very early with computers and I'm very visual in my nature. I like to draw. I actually did comics and book covers and things like that. So that's moved me to graphics, then to computer vision and understanding what's around.

And in some sense, VR and AR also include another love that I have, which is interaction. So suddenly it's not just nice graphics. We're actually inside them. And that opens a lot of dreams of what can you do? What can you change that you didn't have before? Would that be good? Would that be confusing? I started at the beginning saying that we're sort of living in the cage of our body. What if I could have seen from a different point of

view? What if I could look for the back? If someone is coming, what if I can look behind the corner to see if someone is running and I'm about to collide? Would I find it confusing? Probably because I never experienced looking from a different point of view. Maybe I learned to enjoy that and use more. All right. So how did you end up at MSR? So after several startups, I thought that maybe the best thing is to go for research and try to think ahead for the world. And when I thought of research, Microsoft Research was a shining diamond that I immediately thought about.

It has so many great names that I knew that were working there, like Andy Wilson or Bill Baxton or Ken Hinckley. And when I joined in, I actually saw how great it is that you have collaborators that you can do. And all the works that we're talking about is basically a collaborative. People like Mark Gonzalez-Franco and Mike Sinclair are hardware genius or Christian Holt and many others.

And of course, a lot of amazing interns that are coming in and work on it. So I contacted Harry Shum, he was the head of vision in the Microsoft lab in Beijing, and I moved over there. Really? Yeah. So I've been for two years in the Microsoft research in Beijing. And then there was a new group that was forming at Microsoft called Virtual Earth. And I was lucky enough to be a part of that group. And I moved to Redmond. After six years there, I felt like the work that we did was amazing, but the way people experienced it was on a 2D screen. It was obvious that the next step should be immersive.

I love that. I'm waiting. I'm waiting. Well, as we close, I'd like you to give us a vision of the future from your perspective. So if, as you suggest, we're on the cusp of a revolution, what's next? How will you, along with the emerging researchers, get us to the next level in AR and VR? And what does the world look like if you're wildly successful? Oh, good question. And probably a smart person should not talk about the future. But in general, I see a lot of opportunity. We are changing the signal that we send to our senses, which is the way that we are experiencing the world around us, our life. I'm not sure we still understand everything in this field.

And so there's a lot of opportunities here, both on learning what's happening around us so we can better represent that and answer that. There's an effort on how to understand what people want with as little information as possible. When I use augmented reality, the most common mode is nothing.

Right now, most of the time, I don't want any information in my field of view. I like reality. But whenever I need it, I want it. So we do things that I call contracts with the system. For example, if I take a look at the phone or the watch, it has a lot of social implication. You will say, oh, he might need to run somewhere or he might find me boring. Right. But what if I will just gaze and I see in the corner of the walls and the ceiling, what time is it and how much time I have till my next appointment. You will not know about that. And I don't see that all the time, only when I do this gesture that the system knows.

So the more we know about the user, the more we just, you know, fit his wishes or her. And that's a great challenge, as well as how we can use the fact that we're not limited with distances, physical limitations, and so on, to maybe do better communication, better work, and so on. So it sounds really multidisciplinary, what you're talking about. So there's all kinds of places for a variety of kinds of researchers to be engaging in this work. Yes. Maybe it's not even seen like that right now, but I think it's an exciting world coming in when we only see the tip of the iceberg. And it's a virtual iceberg at that.

A. L. Ofeck, thank you so much for joining us today. Thank you. It was fun. To learn more about Dr. Eyal Ofec and the latest in augmented and virtual reality research, visit Microsoft.com slash research.