Microsoft Research Podcast - 020r - Getting good VIBEs from your computer with Dr. Mary Czerwinski

Episode Date: April 29, 2020

This episode originally aired in April, 2018 Emotions are fundamental to human interaction, but in a world where humans are increasingly interacting with AI systems, Dr. Mary Czerwinski, Principal Researcher and Research Manager of the Visualization and Interaction for Business and Entertainment group at Microsoft Research, believes emotions may be fundamental to our interactions with machines as well. And through her team’s work in affective computing, the quest to bring Artificial Emotional Intelligence – or AEI – to our computers may be closer than we think. Today, Dr. Czerwinski tells us how a cognitive psychologist found her way into the research division of the world’s largest software company, suggests that rather than trying to be productive 24/7, we should aim for Emotional Homeostasis instead, and tells us how, if we do it right, our machines could become a sort of “emotional at-work DJ,” sensing and responding to our emotional states, and helping us to become happier and more productive at the same time.

When I spoke with Dr. Mary Cherwinski two years ago, we talked about the vital role emotions play in our daily lives, including our ability to be productive at work, and how bringing emotional intelligence to machines might help us improve our work and our well-being at the same time. Whether you heard Mary's podcast in April of 2018, or you're ready to find out how technology could make you more productive and emotionally resilient today,

I know you'll enjoy Episode 20 of the Microsoft Research Podcast, Ready to find out how technology could make you more productive and emotionally resilient today? I know you'll enjoy episode 20 of the Microsoft Research Podcast, Getting Good Vibes from Your Computer. We were calling it like a DJ. It's your emotional at-work DJ. So, you know, when you need everything really ramped up and going strong, like a DJ knows how to do. But then when it's time to calm the crowd down, maybe our software can learn, okay, Mary needs to take a break. Mary needs to slow down. Mary needs to go get a glass of water. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Husinga.

Emotions are fundamental to human interaction, but in a world where humans are increasingly interacting with AI systems, Dr. Mary Cherwinsky, Principal Researcher and Research Manager of the Visualization and Interaction for Business and Entertainment Group at Microsoft Research, believes emotions may be fundamental to our interactions with machines as well. And through her team's work in affective computing, the quest to bring artificial emotional intelligence, or AEI, to our computers may be closer than we think. Today, Dr. Cherwinsky tells us how a cognitive psychologist found her way into the research division of the world's largest software company, suggests that rather than trying to be productive 24-7, we should aim for emotional homeostasis instead, and tells us how, if we do it right, our machines could become a sort of emotional at-work DJ,

sensing and responding to our emotional states and helping us to become happier and more productive at the same time. That and much more on this episode of the Microsoft Research Podcast. Mary Trewinsky, welcome to the podcast today. It's great to have you with us. Thank you. It's an honor to be here. So you work under the larger umbrella of human-computer interaction at Microsoft Research, but more specifically, you're the principal researcher and the research manager of the Vibe group. Right.

Tell us what Vibe stands for and what gets people in your group up in the morning. Vibe stands for visualization andaction for Business and Entertainment. Most of the people in the group are very deeply into various aspects of information visualization, so helping people work with big data better or building tools for programmers so programmers can deal with all the vast amounts of data we have coming in these days, or in the area of effective computing, which is where I'm squarely putting my research these days. Microsoft Research has a number of social science researchers now.

Yes. But you actually were the first social science researcher. How did you end up here? Did they come looking for you? Who decided they needed a PhD in cognitive psychology to help with computer science research? That's a great question. I took the initiative to meet with Dan Ling, who was managing all of research at that time, and talked to him about social science and the benefits of having social scientists on their research teams. But it was actually Eric Horvitz and George Robertson that actually sought me out as a psychologist

to partner with them in their research around attention and rendering on the screen and redesigning windows in 3D. And so the two of them really kind of helped usher me over to research. I was actually in product at the time, so I was in Microsoft, just wasn't in the research group. So it was a great time to come out into the field as a cognitive psychologist because back then we had one computer screen and it was all about users being able to look at the screen and make sense of it. Well, that's what cognitive psychologists study, perception and cognition around information, learning, memory, making decisions, attending to various aspects of the displays. So it was perfect timing for me to put what I did my PhD in to work

for Microsoft. You know, I would have not even gone there in terms of that thinking, just given where we are today with the attention economy, they're calling it. Yes. And there's so many more demands on our attention. So tell me how it's evolved since when you started to now. Yeah, well, I mean, back then, the experiments we would do, for instance, with the astronauts at Johnson Space Center, I would be looking at their layouts of their displays, and they'd be running nine experiments at a time, and they had to monitor them all the time, in addition to the various space station systems like thermonuclear control, which is kind of important. And so we looked at character-based user interfaces and how people can track that kind of information. That all changed when graphical user interfaces came along. And then it was more about how you grouped things and how you arranged them on the screen. And it's still about that today. That hasn't gone away. But now, you know, everything's more social and now we have multiple devices and multiple displays we have to look at. So it's just fanned our attention out even more broadly and harder to control, obviously. Yeah. So it makes your job a little harder as well.

Yes. But that's exciting. Yeah, it's really fun. of big ideas, emotion tracking, information worker task management, and healthcare and wellness for individuals and groups. And we'll get to each of those in turn. But for starters, let's talk a little bit about this big idea of affective computing and the quest for artificial emotional intelligence or AEI. What is it and why is it important? Well, I don't think we'll actually have quite natural user interfaces until we can actually talk to our systems the way you and I naturally talk to each other. And so a large part of that is these really natural, almost that you're mimicking me basically and my thoughts. So we believe that systems will need to have that kind of emotional intelligence EQ in order for them to be natural enough for us to really engage with them and want to keep engaging with them.

So there's not that freaky uncanny valley where it just doesn't feel natural, doesn't feel right. So we've been pursuing this emotion tracking in order to track your emotion as a user so that the system can respond more like a human would emotionally appropriately. So do you think we'll ever get real artificial emotional intelligence with a machine? Well, that's TBD. I would rather be on the edge exploring those things as they happen as a scientist and trying to make the most of it. Part of your research involves designing what you call delightful systems, but it isn't always easy to tell whether they're really delightful or not. How are you tackling this, Mary? I like to say, how can we say that we're designing systems that delight our users if we don't actually know? So that is why we're using cameras and microphones

to look at our users' faces, to see what their facial gestures are. Are they smiling? Are they squinting their eyes? Are they frowning? But also listen to their voices, hear how they're saying things about working with our systems. And if we can build these kinds of emotion-sensing platforms into all of our products, then we'll know from the get-go if we're actually designing systems and tools that users are delighted by, or are they frustrated, and then we should fix those areas that seem to bring on the frustration. So I think if you build these tools into the platform of all of our systems that we build in the future, we'll know better if we're making systems and software that people love. You were part of a recorded presentation called, And How Does That Make You Feel?

Which I loved the name of. Yes. You know, it's like the psychiatrist. Yes. Does that make you feel? Exactly. And at several points, you got frustrated with the fact that your video wouldn't play. And about the third time, you actually flashed a big smile and go, that's just great. And I remember thinking, how would I as a

machine know that you weren't really happy and delighted with me? That's a great question. That's a great question. Because we never think, now I'm channeling Daniel McDuff on my team, we never believe that a one to 10 second slice of a user's face is going to give us anything worthwhile. What we believe in is that you have to watch and listen to this user for a long time longitudinally so that we really understand the range of a person's emotions. People are all over the place with how labile their emotional state is. I'm pretty up and down. I have a broad range. But some people aren't. Some people are much flatter or just more controlled with their emotions.

And so you really do have to study an individual for a long time before you know. But then also, the machine needs context about what you're doing. So if the machine had been watching and seeing that I couldn't play my video, the machine would learn that that smile is probably a smile of complete frustration. Or a grimace. Yeah, or a grimace. Yeah, or a grimace, a grimacing smile. It could have been. But yeah, so the machine needs the context and the longitudinal data about you, yourself as a person, before it can have really good assuredness

in that it's making the right emotional classification. So that's an interesting thread to drill in on because you want to build a system that will work for many people, not just one person. And if my emotional ups and downs are different from hers, then how are you going to bring the research together and make a machine that can tell the difference if you need to track me for a long time and him for a long, you know, times a million? Yeah, well, because you can do this now with simple webcam and mic on any laptop or PC, it's pretty easy and the user doesn't have to do anything to do this kind of tracking longitudinally. And we're doing it right now here in Building 99. Many of us are getting our emotions classified while we're using everyday software systems and doing our regular

work. So now we have thousands and thousands of hours of these emotions tracked, and we can bring them back to bear, not only with generalized models, but also personalized models that could be more relevant just to me. So we can do both scales quite well. Okay. So you could even employ machine learning AI to actually customize in a particular product for a particular person. Yes. Yes. That's the angle. That would be cool. You framed some of your research in the category of productivity tools. Tell us about the technology you're working on to help mitigate negative interruptions, multitasking, or I think you really would call it task switching because there is no such thing. At least that's what I tell my daughter. You're not multitasking. And some of the general lack

of focus that we experience at work and if we're honest in life. Yes. You know, I do a lot of work with Shamsi Iqbal and Gloria Mark, and Gloria has really opened my eyes to the fact that everybody has a really different personality. Some people, it turns out, are quite adept at managing their workflow. So they stay pretty concentrated and focused for longer periods of time than other people, and they take breaks when they know that they need to, when they need to refresh, because no one can stay focused 24-7. That's a myth. So we like to refer to something called emotional homeostasis, which is you stay focused, you work hard, maybe you are even a little stressed, and then you do something to balance that out, right? So maybe you use social media if you're one of those people. But other people have a harder time staying focused for longer periods of time. And actually, every time they hear a ping or every time they get an email notification, they go to it. You know, the little numbers on the icons are the worst thing for these kinds of people because they see they have something new and they have to go check it.

So for those kinds of people, we can actually use machine learning and this personalized kind of software we've been talking about to see when you need to be in the flow, to see when you actually do your best work and to kind of help you stay there. So perhaps that's the time when you're really starting to get into your work. Perhaps that's the time we turn off notifications or hold less high priority notifications away from you. Turn off that inbox, you know, dinger and let you focus. We might even turn off social media if we go so far. The user gives us permission to do that. We'll do whatever the user wants us to do to keep them in the zone. And then, you know, when you tend to kind of come out of that flow, maybe it's right before lunch, let it all flow back in. It'll be just fine.

And the machine can see that you're moving out of the zone and it's going to turn back on automatically? This is current research we're doing right now. This is our hypothesis that we'll be able to see when you prefer to have your hardcore work moments and when you are okay with letting notifications through and possible work breaks through. Or we can see if you're one of those people that stays really focused. Maybe we can see you've been focused a little too long for yourself. Maybe you are getting a little stressed out and a little walk would be quite welcome at that point, but it's all work that's in progress. So we still don't know how these, we kind of like, we were calling it like a DJ. It's your emotional at work DJ. So, you know, when you need everything really ramped up and going strong, like a DJ knows how

to do, but then when it's time to calm the crowd down, maybe our software can learn, okay, Mary needs to take a break. Mary needs to slow down. Mary needs to go get a glass of water, bio break, something, you know. So we'll see how much users will be welcoming such software and we'll design it until they do welcome it. We'll iterate it and do it better. If we're using these tools to get in the zone and stay there, should we call it artificial productivity? No, it's real productivity. It's artificially assisted. Artificially assisted productivity. Yeah, yeah. Well, so let me ask you this. As a psychologist, what happens to good old-fashioned self-discipline? Have we just lost that in this digital world?

Well, I will tell you, in a study we ran two summers ago, a couple of our participants, we turned off all their social media, all their notifications, they weren't allowed to use their phone for eight hours a day. And a couple of our participants did make note of the fact that they had lost sight of how often they were going to social media and checking their phones. And they were shocked by how much more time they had. So in some sense, the sad truth is, yes, we have trained ourselves to do this task switching on an almost constant basis. And on a great note, software can help. So, you know, maybe we can train it back. Let's get a little meta on that topic and this whole concept of digital assistance and

affective agents. And kind of on the same note, have we really reached the point where we need technology to save us from our technology? I just did a press interview yesterday where I did actually make the claim that we need technology to help us from our technology. So I can stand by that. I've said that. But as I said, all of our tasks switching bad behavior was trained. It was trained by the tools and the features that we ask software companies to build for us or tech companies in general. So we can use the technology to retrain ourselves to focus. I believe we can. I know we can. The research shows we can. And we'll make better decisions. We'll do better at our jobs and we'll be more productive. So yes, if we have to start using technology as training wheels to get rid of all the bad technology that's, you know,

bifurcating and dividing our attention, I don't think that's a bad thing. I think it's a good thing. We've come around full circle. From a psychological point of view, aren't we at the risk of outsourcing some basic human skill sets that are emotional, that are necessary for emotional growth and health? Right. So I wouldn't say that we've gone that far, but it is nice that we can use the machine intelligence to kind of protect us again and get us back into focus. I think that's going to be great. In terms of the emotional side of things, I really do think that the advent of robots taking care of children using robots and personal assistance is on their own communication, growth and behavior. But also I'm really adamant that

machines need to have EQ. Not because I don't want you to know that you're working with a machine or a robot, but because I want that conversation to have an emotional balance and to be emotionally mature so that kids don't grow up with this imbalance, with lack of EQ, lack of real intelligence. So I think it goes both ways, and that's why I want to study both of those aspects of it, because it could be worrisome, right? Absolutely. So on that thread, affective computing, as I'm looking at it, seems to have two sides to it. One is designing machines that can interpret human emotional states and adapt to them. And the other is designing machines that can help humans interpret their own emotional states. First, am I right?

And second, if so, why is this two-sided approach necessary and happening? Most people aren't very aware of their emotional state. And so what we find when we first start doing our experiments, it actually takes people a day or two to actually start understanding how they actually feel. I'm usually in the positive quadrant of how we do these self-ratings. And it took me even a little while to realize I'm not always, you know, happy and high energy, right? Sometimes I am kind of low energy and sometimes I am sad. So after you track yourself for a while and you are honest with yourself, then you start to realize how the states move around. And so in any experiment,

we always know that at the very beginning, users might not be super good at it, but they'll get better at it over time. And I think that it is very useful because we forget our emotional states very, very quickly. Quick little survey we did showed people forget how they were feeling about a day. And if it's a really big event that happened in your life, you're probably going to remember how you felt. It's those little patterns of micro badness and micro goodness we don't actually pay that much attention to. And it could really help us make better decisions. Like for instance, people were telling us after like a week of tracking themselves and whatnot, that they wouldn't remember these things a year from now. So having a system that tracked it would be quite

useful. Now it has to track it accurately. So there should be a way that you can correct the system when it's wrong. But I think it's kind of nice to go back and look at how I was feeling. And actually, I'm always surprised that I am happier most of the time, you know, and frustrated very little of the time. But there are moments you would forget otherwise. So it's useful to you. It might be useful if you were taking care of a loved one to be able to see those tracks. Right now, most people go to a therapist's office or a doctor's office. They fill out a paper form for how they've been feeling the last four weeks. Well, people don't remember how they were feeling the last four weeks. So these tools can be very useful if they could be shared with loved ones and caregivers.

Yeah. So let me ask a question about that. If I'm going to be monitored in all these situations, what's watching me? What's recording? What I want to know is I'm in front of my computer right now. Okay. And it's a laptop. Uh-huh.

And I might be working at a desktop at my desk. But out in the wild, I don't have these devices. Oh, right. No, you would have to have some device that has a camera and or a mic, but you can have either one. Okay. That's constantly listening to you in those particular contexts that you might want it to. Right now, it's very easy when people are just at their desk or in a meeting. So actually, we're tackling meetings next, which will be fascinating. Oh, wow. Yeah. But yeah, you would have to have some kind of recording device on you if you wanted to get it 24-7 all day long.

Well, let's talk about that for a second, and then we'll swing back to some other topics. Increasingly, the mechanisms that are collecting data include video, audio, maybe wearables, ingestibles, implantables, lunchables, I don't know. How do we reconcile the desire to have this knowledge, this quantified self, this self-awareness with our desire for privacy and our fear of big brother? Great. No, that's why we do everything we can to keep your signal private to you. There's no identifiable information that goes into the cloud at all. We as researchers can't go in and look at your data if you're running our system. We can't. And so is it encrypted? I mean, how would I be confident? You are completely de-identifiable. We can't find you unless you give us a code back,

you know, your ID and tell us you want your data, which of course we can give you if you want it. That's the only way we know who you are. Well, I just interviewed Kristen Lauder, who works in cryptography. And it's like all of the different teams that come together to make these tools both fantastic and trustworthy at the same time is super important. And we will surely use her stuff. We work with Ronnie on her team, so he talks to me about what they're doing all the time. It'll be a good thing to have. Yeah. In fact, you used a funny phrase to frame the human-computer interaction group's relationship to other groups at Microsoft. Tell us what it is and why you say it.

Yeah, human computer interaction people often play the glue between various technology teams, because we're usually the user facing part of the experiment. And we can see when, you know, there's a technology that's very right to present to a user, but it needs something. Like we were just talking about, it needs encrypted privacy. So we'll bring the two teams that need to talk to each other together because we go out and put our user interfaces on all of their technologies. So we kind of know what they're doing. So that's why I say that. Sure. So tell me what's going on in the research stage and even in the product stage on things that can help me track my emotions or my state of being at work. What we do know is that users really could use help from our technology to turn off notifications, to possibly turn off non-work-related websites

like social media, and help the user focus. And maybe the user just wants to say something like, I just want to turn all that off for a half hour. Then I'll be fine. And the system should help them do that. Eventually, the goal would be to do that automatically for the user. So the user doesn't have to remember to do something manually on their own. That's an interesting thing. I don't know if it's better if I develop the skill to remember to turn it off or to have somebody do it for me. We have to do studies on all of this. Right? Right. One of the biggest problems I have, and I know I'm not alone, is wrangling and making sense of my data.

How are you helping me? Ah, well, I know you did a podcast of Stephen Drucker. I did. He was so cool. That does wonderful work in information visualization. But we have many researchers that do work in information visualization at Microsoft Research. They're all fabulous. They all take different aspects of how you look at your data. Some of them like to work with networked data.

Some of them like to work with time-based data. So there are many, many tools in the InfoViz community. InfoViz stands for information visualization. Thank you. You're welcome. That can be thrown at these kinds of big data problems. Yeah. Another thing that Danielle Fisher has done in our group, which I thought was really nice, is he starts to show you your data as it's coming in. But if there's so much data that it can't all be rendered at once, it just gives you a feel for what the data is going to look like. And you decide when you think you have enough to make a decision. So they're all working with these kinds of interactive visualization tools to help you see patterns in your data that you might not have been able to see in an Excel spreadsheet using standard bar graphs or line graphs.

So you said some of this is still in the research stage, but some of it's already incorporated into BI and Excel and things like that. Yes, Power BI has been a great vehicle for taking visualization work and exposing it to end users. And I think it's been very helpful. It's very hard to come up with novel visualizations. So it takes a lot of creativity, a lot of work to cook up something that people can use pretty quickly because they're usually pretty sophisticated versus the standard bar chart or pie charts. So it does take users a little while to learn how to use them, but then when they do, they're very powerful and useful. Yeah, yeah. Let's look to the future for a second. Your group is publishing prolifically. You're going to CHI in a bit with a lot of papers.

What excites you most about what the field is doing right now and what might be exciting to the next generation of human-computer interaction scholars? Right. Well, I think the whole movement towards the gig economy is really new and exciting, and it's studied pretty hard at CHI. It's represented pretty well at CHI. So I tend to go into two tracks when I'm at CHI because there are like 14 parallel tracks. So you have to pick. And I tend to go into mental health and e-health. And then I try to go to the gig economy tracks because that world is moving so fast. So the advent of micro tasks are emerging. You know, even just think of writing a Word document now. We have researchers in the building who are working on just taking little

parts of writing that document and making it a task for somebody to do out there, right? Jamie Teevan and I did a whole podcast on this. Okay, good. And Shamsi Iqbal is doing it as well as some others. But that economy, whether you're talking about taxi drivers to airplane designers, that economy is going gig. And so it's just fascinating to listen to hear the research they're doing and the trends that are happening and how those gig workers actually work with each other to make sure each other succeed and they make a good wage. I think the young guns will need to do a lot of work on the gig economy. So that's exciting. But also health and mental health. I mean, really moving fast and using machine learning to look at, you know, precision psychology, for instance, is a huge topic. So I'm hoping the young people look at those two areas.

But there are many more very hot areas. Those are just my two favorite. We talked about what gets you up in the morning. Is there anything particularly that keeps you up at night, Mary? Yes, I do get worried about our systems that will eventually have good EQ, but they don't yet. And how working with the systems today when they're not truly intelligent, they don't filter by age, gender, location, et cetera, in terms of what they say pretty much these days. I worry that it changes people's communication patterns because they're going to pattern off the system. So I really get worried in particular about young people. The elderly, I imagine their morals are already built in and they've learned their communication

style pretty well by now. But I really worry about the young generation and the generation that maybe, as we said, maybe they're going to be working more with a robot than they are with, you know, a teaching assistant in the future. So I want to make sure that that modeling is what we as a society see as appropriate, moral, and positive. So I really want to focus on generational issues like that. And also what is the appropriate EQ for a machine, right? Do you want to know that it's not human? I think most of us want to know that it's not human. And so how do you do that and yet still make the conversation feel natural and make the conversation feel appropriate? so profound because what much of the work happening here is about is making computers more human-like. And yet, when you ask that question, do you want to know? I'm nodding my head really hard. Yes, I want to know it's not. I don't want to be fooled by the Turing test. Right, right. So we have to, again, as a society, come up with a plan, basically. Are you doing that?

Thinking about it hard. If you weren't a researcher, what would you be? I really wanted to go back and get my PhD in biopsychology at one point. I, you know, I can't not be a researcher. I'm sorry. I was just going to say, you answered that by saying I'd be a different kind of researcher. I mean, if I could be a professional tennis player, I'd do it, but I don't have the talent, so it's not going to happen. Last question. What advice would you give to your 25-year-old self? This is for the researchers out there who would be looking at, what do I do? Where

do I put my time, talent, treasure? Yeah, what I tend to tell young people that I mentor that are still in graduate school, for instance, because that's about 25, I tell them not to think that life is just a straight line, right? It's never a straight line. You don't get your PhD and go from point A to point B to point C to point D. Sometimes if you stay in academia, that can happen. But even then, I don't think your career is going to go in a straight line. Too much happens. And in my particular case, technology changes way too fast. So you always have to be open to something. I thought for sure I was going to go into academia. I couldn't because of a two-body problem. I ended up in industry. Oh my God, was that the best thing for me ever? That was just perfect. And then I jumped around, right? I went to Johnson Space

and I went to Bell Communications Research. I kept my foot in academia because I thought, you know, someday I might want to teach again. Got this wonderful job at Compaq, which led me to Microsoft, which led me to Microsoft Research. You know, you just got to stay open and you'll know when it's time to leave. Things won't feel right. And then that great opportunity might be out there and just be open to it. Mary Cherwinski, thanks for joining us today and sharing so much about what you're doing. And we can't wait to see what's going to happen. Thank you. It was my pleasure. To learn more about Dr. Mary Cherwinski and how to have a better relationship with your computer, visit Microsoft.com slash research.