Microsoft Research Podcast - 004r - Getting Virtual with Dr. Mar Gonzalez Franco

Episode Date: April 15, 2020

This episode originally aired in December, 2017 On today’s episode, neuroscientist and virtual reality researcher, Dr. Mar Gonzalez Franco, talks about her work in VR, explains how avatars can help increase our empathy and reduce our biases via role play, and addresses the misconceptions that exist between the immersive experiences of virtual reality and psychedelic drugs.

Dr. Mara González-Franco was one of the first guests on our podcast, and she's still one of the most intriguing. Back in December of 2017, she shared her unique vision for virtual reality and its potential to impact human perception and even change the way we live. Whether you were with us at the beginning, or you're just exploring Mara's world of virtual reality for the first time, I know you'll enjoy Episode 4 of the Microsoft Research Podcast, Getting Virtual. Imagine you enter this new virtual body that you look down

and your hands are these other hands. They move as you move perfectly, only you're a different race or a different gender. So you can experience the life perspective of a different gender. So you can experience the life perspective of a different person from within. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Huzenga.

Today, we're here with neuroscientist and Microsoft researcher, Dr. Mara Gonzalez-Franco. She'll discuss her research in virtual reality, the importance of avatars, and she'll clear up the misconceptions between VR and psychedelic drugs. That and much more on this episode of the Microsoft Research Podcast. Mar, you have a really interesting background. How did you end up doing research at Microsoft? I think in life, there are sort of cycles that you end up doing the same thing again. Like I started trained as a computer scientist, mathematician, and then I became very interested by neuroscience. And then I came back to computer science. So it kind of comes together. I would say there is always not a single route to get somewhere. And definitely you need to take it step by step.

So I wouldn't be here if I didn't do my PhD with Mel Slater and partially at the MIT, for sure. I mean, this Microsoft research really scouts for talent in the wide spectrum on a global scale. And you have to be there. And there is a part of being lucky to be in a very hot spot that is of interest to the company in a very specific moment. You told me that VR can trick the brain.

Yes. And that people know it's fake, but they have a strong experience of reality anyway. So what's good about that? And what's bad about that? And what do we take away from that? It kind of scares me, but it's kind of cool. So yeah, it's very interesting because we are providing a real-time stream of information to our senses.

Mainly virtual reality has been based on visual. And humans are very visual animals. We believe what we see and virtual reality provides that. It provides a congruent sensory motor stimulation. One of the experiments I did, you put on some sort of glasses, an HMD, with virtual reality. You put your hand on top of a table and you see a virtual hand instead of your hand. And that hand gets attacked. And what I find is that the motor cortex gets activated in the same way that it would be activated for a real attack to your real hand. So you really perceive this is your hand. And I think this is

very interesting because it means not only that you can enter a virtual reality, you can change your body in that reality. So what are the implications of what you just said in terms of application outside the fact that you've observed that you can make people believe that a virtual hand is actually their hand? Some very interesting applications have to do with empathy. Imagine you enter this new virtual body that you look down and your hands are these other hands. They move as you move perfectly, only you're a different race or a different gender.

So you can experience the life perspective of a different person from within. Philosophers and psychologists have explored this role-playing. This is role-playing to the maximum exponential. You're basically within a different body. You're talking about avatars now, whether it's just a hand avatar or a full body avatar. And that's the main component of virtual reality. I think avatars are becoming more important than ever with virtual reality.

We experience reality and people experience ourselves through our bodies. And so it's very easy to evaluate things from the perspective of our own body. And in virtual reality, we need to have a body, an avatar that represents us. So if we can have realistic experiences through avatars, what other things might we be able to do through VR? How else might it help us? This is a very important topic in my research. The avatar that you embody.

If you embody an avatar that is of a different race, you can reduce your racial bias. And this is some experiments that have already been done. There are other experiments that have shown that if you embody an avatar that is older, so it looks like you, it's a lookalike avatar, just a little bit older, your financial decisions after you come out become more conservative. You're more likely to save money. And not only this, even more traditional fears like fear of height, fear of flying, virtual reality provides a very controlled platform to perform therapy interventions. And then during an intervention, people might

develop a crisis and you might want to take them out. In virtual reality, it's simply a headset. You remove it. You're in an airplane, you're out from an airplane. If you want to treat somebody who's a fear of flight, who has this problem, by putting them inside a real plane and flying them to New York, they might develop the crisis in the middle of the flight and there is no way for you to stop it. So in that scenario, it's much more secure. So for anybody who's treating this kind of phobias, virtual reality is clearly a very good option. So is this happening already or is this just something that you're looking at one of the possible applications for VR in the future? No, people are already using it. Of course, there are only few people that have these really, really extreme

phobias that cannot be treated in a less immersive way. Like you could treat your phobias by going in the real world rather than in the brittle board, but it also allows you to treat your phobias from home. You have a device at home. You don't need to go to a center to be exposed to this therapy. So it also democratized a lot these sort of treatments or interventions. Let's talk about the illusory nature of VR for a bit. I read an article where the headline said, and I quote, Microsoft says virtual reality could make you hallucinate in the same way as LSD. And that was a rather sensational version of what you actually said, I think.

But can you talk a bit about how your work in VR is like and even unlike hallucinogenic drugs? When Jaron Lanier started the first virtual reality company, he was faced all the time like, oh, is this going to be like an LSD trip? And of course, the answer is not. But I'm going to go a bit more in detail because I do think it's legitimate for people to be concerned about how a technology is going to affect their life. So virtual reality, in the best form, provides a very strong illusory setup to create illusions. Illusions such as there is a glass on top of this table. But in reality, there is not such thing when the reality has the same power. But this is an illusion in which you never lose

your higher cognitive functioning. There is no chemical alteration of your brain. It's purely sensorial. You're seeing this glass of water, it looks super realistic, and you move around and the glasses are staying there. So at the end, it really feels like there is something there. But you take out the glasses and there is nothing there. Compared to drugs, this person has never lost their higher cognitive function during this illusory state. And that changes completely the topic because higher cognitive function is the one that allows you to make conscious decisions, decision-making processes, what's right, what's wrong. And because that's never altered, you're always aware that this is

not real versus when people do drugs, that's completely lost. So I would say that these are very two different ways of experiencing reality. You're really interested in the societal impacts of virtual reality. So what are the best things about your work in VR? First of all, this democratization of the technology, arriving people who have never been able to own any previous technological advancement, and now all of a sudden they have something that includes them all. You know, it's like one ring to rule them all. And augmented reality and virtual reality are getting there.

In fact, there is still a bit the fight on whether it's going to take the form of a phone that you put on top, or it's going to be a completely new device that you can render a phone in it. But it's true that it's going to be mobile in the sense that the battery, it's going to be there and you're going to put it on and render from first person perspective. So that's very exciting because it's going to have very big implications on society. It's going to change many things of how we live. Give me an example of what you think that might be, like one thing that could help us? Yeah, I think in particular, this tendency we have to buy so many gadgets, that might actually go away. And we have one single gadget.

This has not happened with any other device. Even we have tablets now, we still have a laptop or we work on a desktop. If we're talking of a device that is going to be able to embed them all, you're really going to not have a desktop afterwards. You're just going to have this device. So that's because you can render a computer in it, or you can render a tablet, or you can render everything. Explain to me render. How does that work in virtual reality? In virtual reality, you have a display, a near-eye display, because it's like a glass.

So inside these glasses, you digitally print an image, just like you would do in a regular screen. It's just that this screen, it's always available for you. So inside this screen, you can print different things. Printing in digital is usually referred as rendering. What I hear you talking about is like a Google Glass kind of thing. It's very different because Google Glass was an annotation device. It would provide you information on the top, which

is not blended with reality. This is not an annotation technique. This is a very different technological system. And this is with current HoloLens. I can only imagine when we move forward, it's something more like, instead of having a computer at home or a TV, I have my glasses know that this wall has a TV on it.

And when I go home, the TV is there. You know, I don't need to put it back. But you're looking through a device. You're looking through a device like glasses, normal glasses. So does everyone in your family have to have them? I feel like you have a family of five. Everyone has to have the glasses in order to see the TV, the virtual TV. We're working on this because it's interesting to explore what each person will see. Because also using these glasses, I could do something very unique, which is seeing what you're seeing.

If we're sitting one in front of each other and we have a tart in front of us, you're seeing one perspective of the tart and I'm seeing another one. What if we both could see exactly the same thing despite being on different positions? So, I mean, we start a very interesting topic of perspective. Wow. My mind is just reeling with what you're talking about and trying to imagine because my paradigms are so right now, right? My way of writing is using a keyboard and a screen, but you're suggesting a whole new paradigm that we don't even imagine. If we do it well enough, you will have to change nothing from your current interaction techniques. They will be embedded in the new technology.

Let me just extrapolate here. I could have a device on and render a keyboard. Yeah. Or you will take your pen, physical pen, the pen will be recognized by these glasses, and you'll write on a digital notebook. This is so sci-fi. Science fiction is a very good inspiration for science. What's the biggest limitation you face right now? Science.

What's the biggest limitation you face right now? You're sitting in Redmond doing research. And if you could say, hey, this would make my life way easier, what I'm trying to figure out here. Well, what kinds of things are impeding your progress? I think Microsoft is very good at giving all the resources you need to do whatever you want to do. Of course, you have to deliver, but that's like in every other job. So I don't think I have many, I would say it's time. I don't have time to do everything I want. But this is also

good because it gives you the unique opportunity to focus. You need to focus and you need to pick fights that you want, the most important ones. And you actually said at one point that virtual reality can prove neuroscience. Yes. So there are some theories that are only available because they have explored very particular pathologies that happen very rarely in the normal population. Maybe one of a million people have something like that. Maybe it's something related to an epilepsy removal of an epilepsy center that has detached your frontal lobe from the motor cortex, for example. So this is very unique scenarios. And there are people who undergo this pathology of the anarchic hand illusion,

which they cannot control their hand at will. They know it's their hand, but they cannot control it. And this is a real thing in real life. Yes. Anarchic hand. Yeah, it's of one hand only. Yes. No, but it's like simple things like you're eating a sandwich. I'm hungry. My hand knows I'm hungry. It picks your sandwich. So these people learn to, because the other hand usually is fully functional, they learn to hold their hand. So they just hold themselves. So they keep it from grabbing my

sandwich. Yes. For example. Is that for real? Yes, it is. This they keep it from grabbing my sandwich. Yes, for example. Is that for real? Yes, yes, yes. This is a real pathology. Okay. But this is a case study type of thing. Okay. If you want to develop a real model of this,

you might want to try to produce the same kind of illusion on healthy individuals. And then you can see what happens in their brain when this happens, and then compare it to what happens to a person who has this real pathology. Of course, these healthy participants, once they go out from VR, they are perfectly well. They don't have any issue on their control of their body. Just like when they enter virtual reality, they cannot control their body as they wish. But the point would be to see what comparisons you could make in a healthy brain versus an unhealthy brain and how that might be able to treat this.

Yes, and also to understand how the brain works in general for both healthy and unhealthy. So there are two mechanisms in the brain that we use to control our experiences. So the first one is a bottom-up. It's purely sensorial. Whatever we feel, we touch, we see, we integrate it in a multi-sensory way and that's how we define reality. The other one is we have predictions, and sometimes they're very strong. For example, when we feel the phone vibrates and it didn't. This is a top-down illusion. Other people experience that too, besides me?

Yes. I'm not kidding. Yeah, it's called the… Phantom phone syndrome. Phantom phone, yeah. Yes. Get out. Yeah, so this experience is very sensorial. You would swear that you felt it. This is how powerful our brain is into manipulating our sensory input. And the interesting thing about

the phantom phone is that we become aware of it, but there are millions of predictions that are being rejected without even us being consciously aware of it, but there are millions of predictions that are being rejected without even us being consciously aware of it. Wow. So in virtual reality, we can kind of start separating both and seeing how they work one with the other. And it's very interesting because when you play with this, you learn a lot about the neuroscience on things that are very hard

to test in reality. So I think virtual reality is definitely a tool for neuroscience. So let's say that you're this generation of virtual reality and neuroscience exploration, if you will. What's on the horizon for students coming out of university now? We have both in the U.S. and in Europe, very important big projects. There is the Brain Project in the U.S. and the Human Brain Project in Europe, both of them with over a billion dollar funding, which is huge. They are mobilizing a whole community to finally get to understand the brain. And still, we don't know it. And certainly, even the people who are

doing AI are very interested on understanding because if you want to simulate something, you need to know how it works. What would you like people to know about the research you're doing? I think in general, one of the things from a researcher perspective that is different from a product developer perspective is the outreach. So the fact that we can communicate that you can do these type of things or you could use technology to explore neuroscience can improve dramatically a whole community of science. So how from one science you can transfer to another. So you are kind of becoming that bridge. I think when we work in research, we can transfer something, show that it's useful for something else, and then empower a whole community into using it in a different way. How does doing research at Microsoft differ from doing research at, say, a university or another academic setting?

The way Microsoft Research moves is very fast compared to academic research. So for me to pursue an idea, I need to convince my VP. So I need to pitch it. I have to think about it. But I don't need to write a grant and then ask this very big organization of grant funding to evaluate it. And so it's much faster. So my ability to reach to people and to collaborate with them, it's incredible. And I love

it. Just as we go, just say one quick thing about what you're the most passionate about right now in your work. I mean, aside from your, you know, outside life. I'm not very interested about producing tactile sensations out of the body. I know this sounds very weird, but I'm creating this system in which you're holding a virtual object and you feel the virtual object being touched, but not directly on your hands, you know, somewhere else. It's kind of difficult to explain it here, but this would mean that we can perceive touch in a similar way that we're perceiving audio. And this is... I'm working with Christopher Berger on this.

He's Caltech. And we're both super excited about it. And this has been my work during the summer. And this has a lot to do with haptics. So if you have a very simple vibrato in your hands and you're able to perceive touch across the space, that's even better what you can do in reality. Mar, you are fascinating.

Absolutely fascinating. I want to have. Mar, you are fascinating. Absolutely fascinating. I want to have happy hour with you for real. Good. Anytime. To learn more about virtual reality and Dr. Mar González-Franco, and the latest in Microsoft research, explore our website at Microsoft.com slash research.