Microsoft Research Podcast - 060 - Empowering people with AI with Dr. Cecily Morrison

Episode Date: January 23, 2019

You never know how an incident in your own life might inspire a breakthrough in science, but Dr. Cecily Morrison, a researcher in the Human Computer Interaction group at Microsoft Research Cambridge, can attest to how even unexpected events can cause us to see things through a different – more inclusive – lens and, ultimately, give rise to innovations in research that impact everyone. On today’s podcast, Dr. Morrison gives us an overview of what she calls the “pillars” of inclusive design, shares how her research is positively impacting people with health issues and disabilities, and tells us how having a child born with blindness put her in touch with a community of people she would otherwise never have met, and on the path to developing Project Torino, an inclusive physical programming language for children with visual impairments.

Working in the health and disability space has been a really interesting space to work with these technologies because you can see on the one hand that they can have a profound impact on the lives of the people that you're working with. And when I say profound, I don't mean, you know, they had a nicer day. I mean, they can have lives and careers that they couldn't consider otherwise. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Huizenga. You never know how an incident in your own life might inspire a breakthrough in science, but Dr. Cecily Morrison,

a researcher in the Human-Computer Interaction Group at Microsoft Research Cambridge, can attest to how even unexpected events can cause us to see things through a different, more inclusive lens and ultimately give rise to innovations in research that impact everyone. On today's podcast, Dr. Morrison gives us an overview of what she calls the pillars of inclusive design, shares how her research is positively impacting people with health issues and disabilities, and tells us how having a child born with blindness put her in touch with a community

of people she would otherwise never have met and on the path to developing Project Torino, an inclusive physical programming language for children with visual impairments. That and much more on this episode of the Microsoft Research Podcast. Cecily Morrison, welcome to the podcast. Thank you. You're a researcher under the big umbrella of human-computer interaction in the Cambridge England Lab of Microsoft Research, and you are working on technologies that enable human health and well-being in the broadest sense. So tell us in the broadest sense about your research.

What gets you up in the morning? I like technologies that helps people live the lives that they want to live. And whether that's because they have a health issue or a disability, or they're just trying to live better, I want to be part of making those technologies. We have quite an exciting group structure that we work in here. So the moment we sit on a floor of multidisciplinary researchers that mix human-computer interaction, design, engineering, software engineering, hardware engineering, we sort of sit together as a community. And then we work across three strands, the future of work, the future of the cloud, and the empowering people with AI. And through those themes of work across the lab, we get to work with people in many different kinds of groups. I specifically work with people

in the machine learning team and looking how the kinds of machine learning opportunities that we have now can underpin experiences that really enable people to do things they couldn't do before. I want to drill in on this idea of inclusive design for a second. It speaks to a mindset and assumptions that researchers make even as they approach working on a new technology. How would you define research that incorporates inclusion from the outset? And how might we change the paradigm so that inclusivity would be the default mode for everyone? So inclusive design, as it's been put through the inclusive design handbook done by Microsoft, has three important pillars.

The first one is to recognize exclusion. So it used to be that disability was a thing that if you had a different physical makeup, you were missing an arm, you couldn't see, you were considered to have a disability. And the World Health Organization changed that definition some years back now to say that actually what disability is, is a mismatch between a person's physical capabilities and the environment which they're in. So if you're a wheelchair user and you don't have curb cuts, then you immediately feel disabled because it's really hard for you to get around. You know what? If you're a buggy user, you feel the same. Somehow you have to get that massive buggy across the pavement. And thank goodness we have curb cuts that were pioneered for people who are using wheelchairs. I think in that regard, as we think about as technologists,

we are people who can recognize and address that exclusion by creating technologies that ensure that there isn't a mismatch between the environment, i.e. the technology people are using, and their particular physical makeup and needs. So I start from that perspective that we as technology designers have an important role to make the world a more inclusive place because it's not about how people are born or how they what happens to their bodies over their lives it's about the environments that we create and technology is an important part of the environments that we create so the second part of inclusive design is really about saying that when we design things we need to design for a set of people and often we implicitly do this by designing for ourselves.

We just don't recognize that we're designing for ourselves. And if we don't have very inclusive teams, that means we get the same ideas over and over again. And they're a little bit different and a little bit this way, a little bit that way, but they're really the same idea. When we start to design for people who have a very different experience of the world, which people with disabilities do, we can start to pull ourselves into a different way of thinking and really start to generate ideas that we wouldn't have considered before. So I think people with disabilities can really inspire us to innovate in ways that we hadn't expected. And the third thing is then to extend to many people. So if we design for a particular group, people say, oh, well, there aren't very

many of them. And, you know, where's my technology? But actually, the exciting thing is, by designing for a particular group who's different, we get new ideas that we can potentially extend to many people. So if you think about designing for somebody with only one arm, and that means, for example, using a computer, a phone, any technology with a single hand, you can think, well, there aren't that many people who only have one arm. But then you start to think, well, how many people have a broken arm at some time in their lives? Well, that's a much larger number.

So that person has what you might think of as a temporary disability. And then what about those people who have what's called a situational disability? So in a particular situation, they only have access to one arm. So I know this quite well as a mother of a small baby. If you have to hold a baby and do something on your phone, you need to do it with one hand, I can guarantee you. So this inclusive design is a way of helping us really generate new ideas by thinking about and working with people with disabilities, and then extending them to help all of us. So we create more innovative technologies that include more people in our world and help us break down those barriers that create disabilities.

Let's talk about this idea of human health and well-being being central to the focus of your work. Even Christopher Bishop at your lab has said healthcare is a fruitful field for AI and machine learning and technology research in general, but it's challenging because that particular area is woefully behind other industries simply in embracing current technologies, let alone emerging ones. So how do you see that landscape given the work you're doing and what can we do about it? Well, I remember when I arrived at Microsoft Research, I was really excited to come here because I just spent four years working in our national health service in the UK, really trying to help them put into

practice some of the technologies that already existed. And man, was it hard work. It was incredibly important work, but it was really, really hard work. And I don't think it's because people are afraid of technologies or they don't want to use technologies, but you're dealing with an incredibly complex organization and you can't get it wrong. You can't get it wrong because the impact you could have on someone's life is beyond what I think we would ethically allow ourselves. So I was excited to come to Microsoft Research when I said, you know, I really want to work on technologies that impact people. little bit more space to be able to experiment and think about new ideas without being so constrained by having to deliver a service every day. One challenge with healthcare is the easiest way to think about what a technology might do is to imagine what people do now and think,

well, how would a technology do that? But actually, that's not really where we see innovation. We see innovation usually coming in at making something different, making something new, or making something easier, not doing something the same. Let's talk about some of your specific research. I want to begin with a really cool project called AssessMS. Tell us how this came about. What was the specific problem you were addressing? And how does this illustrate the goal of collaboration between humans and machines?

Right. So AssessMS was a project to track disease progression in multiple sclerosis using computer vision technology. It was a collaboration between Microsoft Research and Novartis Pharmaceutical with the branch based in Basel, Switzerland. And it really came about as healthcare is moving into the technology space and technology is moving into the healthcare space with these two large companies thinking about what could we do together? How can we bring our expertise together? We were approached by our partner, Novartis, and they said, we would like to have a neurologist in a box. And it took a lot of time of working with them, negotiating with them, doing design work with them to understand that a neurologist in a box is not really what technology is good at, but we could do something even more powerful. And what that something was, was that we

were looking at how do we track disease progression in multiple sclerosis. Now, patients with multiple sclerosis might have very, very different paths of that particular disease. It could progress very quickly and within two years, they lose their lives. They could have it for 60 years and really have minor symptoms such as very numb feet or some cognitive difficulties. These are very, very different experiences and can be very difficult for patients to know when or how or which treatments to start if you don't know any sense of how your disease might progress. And one step in helping patients and clinicians make those decisions is being able to very consistently track when the disease is progressing. Now, that was really difficult when we started because they were using a range of paper and pencil tools where a neurologist would look at a patient, ask them to do a movement such

as extending their arm out to the side and then touching their nose and then checking for tremor in the hand. Now, in one year with one neurologist, they might say, oh, well, that's a tremor of one. In the next year with the next neurologist, they might say, oh, that's a tremor of two. Then there's the question of, has the patient changed or is it just that the neurologist is at a different time and a different neurologist? Because there's no absolute criteria for what is a one and what is a two. And again, if you're lucky enough to have the same doctor, you might be slightly better. But again, it's been a year's time between the two experiences. But what a machine does really well, they're not very good at helping a patient make decisions about their care, but they are very good at doing things consistently.

So tracking disease progression was something that we said, well, we can do very consistently with a machine and we then supply those results to the patient and the neurologist to really think through what are the best options for that patient that particular year. So how is the machine learning technology playing into this? What specific technical aspects to this Assess MS have you seen developing over the course of this project? There are quite a range of things, actually. In the first instance, we were using machine learning to do this categorization. So at the moment, neurological symptoms in MS are already categorized with a particular tool called the Expanded Disability Status Scale, the EDSS. And we were attempting to replicate those measures as being measures that

the clinical field was already comfortable with. And so in that regards, we were using a set of training data of 500 plus patients that we had collected and labeled and using that to train algorithms and test out and research really were more testing out different kinds of algorithms that might be able to discriminate between those patient levels but actually what we did in the human computer interaction side of things was actually making a lot of that machine learning work so the first thing that we needed to do was design a device that helps people capture the data in a form that was standardized enough for the machine learning to work well. The first thing that we saw when we just did a little bit of pilots that the cameras

were tilted, people were out of the frame, you couldn't see half their legs because they had sparkly pants on. There are all kinds of things that you just don't imagine until you go into a real world context that we had to design. And what's, I think for me, quite interesting is that people are really willing to work with a machine so that the machine can see well, as long as they understand how the machine is seeing, and it's not seeing like a person. So we built a physical interface as in a physical prototype, which allowed people to position and see and adjust the way the vision was seeing so it could capture really good quality data for machine learning. Right. That was step one.

And then step two is like, oh, we need're trying to increase our consistency about clinicians, if we use the current way of clinicians label data at the moment, we're going to get the same level of consistency as clinicians. So we won't really have achieved our goal. So you have to come up with a new way to get more precise and consistent labels from clinicians. And again, we did something pretty interesting there. Partially we used interaction design features. So we went with the idea that clinicians and people generally, they're much better at giving relative labels. So this person is better than that person, rather than saying this person is a one and

that person is a two, which we call a discrete label. So what we did is we did a pairwise comparison. We said, okay, tell us which person's more disabled. This worked really well in terms of consistency, although we nearly had all of our clinicians quit because they said, you know, this is incredibly tedious work. And again, that's where machine learning and good design can come in because they said, well, actually, we have this great algorithm called TrueSkill. This is an algorithm that was originally used for matching players in Xbox games.

But actually what it does is give us a probabilistic distribution of how likely someone is better than someone else. So it takes a problem, which is pairwise comparison, which is an n-squared problem, and makes it a linear problem. And to interpret that for people who don't really work in this space, that basically means if you have 100 films to label, that takes you 100 times however long it takes, which in this case is about a second, rather than taking 100 times 100, which is a much longer time. By using sort of thoughtful ways and other kinds of machine learning, we could actually make that process much faster. So we managed to show that we could get much more consistent and finer-grained labels much faster than the original approach.

So we went to build the big system, but in the end, actually, we spend a lot of our time on these challenges that just make computer vision systems work in the real world. Is this working in the real world, or is it still very much at the prototype research stage? Well, I think it was a very large project. A lot of data was collected. The data sets are still there. But what we found was that really the machine learning isn't really up to discriminating the fine level of detail that we need yet. But we have a data set because we expect in the next couple of years it will be.

So it's on pause. Let's talk about one of the most exciting projects you're working on. And it's launching as we speak, called Project Torino. And you said this was sort of a serendipitous, if not accidental, project for you. Tell us all about Project Torino. This is so cool. So Project Torino is a physical programming language for teaching basic programming concepts and computational learning skills to children ages 7 to 11, regardless of their level of visions, whether they're blind, low vision, partially sighted or sighted, it's a tool that children can use.

And it was indeed a serendipitous project. We were exploring technology that blind and low vision children use because we have a blind child. At the time, he was quite young. He was about 18 months. And we really wondered how many blind and low vision people were involved in the design of this technology. And we thought, what would it look like if these kids, these blind and low vision kids that were in our community that we now knew through our son, what would it look like if they were designing the technologies of tomorrow? Their own technologies, other technologies? So we decided to run an outreach workshop,

teaching the children in our community how to do a design process and how to come up with their own ideas. So we brought them together. We had a number of different design process activities that we did. And, you know, they came up with amazing things. We gave them a base technology based on Arduino that turns light into sound. And we just walked them through a process to create something new with that. And they came up with incredible things that you'd never think of. So one young girl came up with an idea of this hat, very, very fashionable hat, I have to say, which adjusted the light so that she could always see because she had a condition where if the light was perfect, she could see almost perfectly. And if the light was just a little bit wrong, she was almost totally blind. So it was quite difficult for her in school. We had another child who created this, you might call it a robot, which was running around his hundred room castle, which was imaginary, I learned in the end, to find out which rooms had windows and which rooms didn't have windows. Because at the age of seven, he had told me very confidently

that his mom had told him that sighted people like windows and he should put them in the rooms with windows. So we were really excited about how engaged the children were. The ideas they came up with was great, but it was an out-of-the-way workshop. So when we were finished with the day, we thought we were finished. And that week, a number of the parents phoned me back or emailed me and said, great, you know, my child has come up with several new ideas. They really want to build them, so how can they code?

And I thought, gosh, I have no idea. Most of the languages that we would use with children of that age group, between 7 and 11, are not very accessible. They're block-based languages. So I asked around, did anybody know? We tried a few things out. We tried putting assistive technologies on existing languages, not very accessible. They're block-based languages. So I asked around, did anybody know? We tried a few things out. We tried putting assistive technologies on existing languages, and we discovered that this was a big failure. The first time I made a child cry, I was a little bit sad, a little bit depressed about that. So that was definitely not the right direction.

But I was having lunch one day with a colleague of mine who works in my group as a hardware research engineer and I said you know is there anything out there that we could hack together just to enable these kids to learn to code give them the basis before they're ready to code with a text-based language with an assistive technology when they're a bit older the answer was well not really but actually I think we can build that. I think we've got a bunch of the base tech there already. So we got a bunch of interns together and off we went. And where is it now? It's been a very exciting journey from that first prototype, which is really a good prototype,

tested with 10 children, to a second and a third prototype, which was then manufactured to test with 100 children. And after an incredibly successful beta trial, we are partnering with American Printing House for the Blind, who will take this technology to market as a product. Wow. How does it work? How does it work? It's a set of physical pods that you connect together with wires.

And each of these pods is a statement in your program. And you can connect a number of pods to create a multi-statement program, which creates music, stories, or poetry. And in the process, with different types of pods, we take children through the different types of control flows that you can have in a programming language. And so this is not just, you know, the basics of programming languages, computational thinking and sort of preparing them, as you say, for what they might want to do when they get older. Yeah, so I think whether children become, you know, software engineers or computer scientists in some way or not, a lot of

the skills that they can learn through coding and through the computational learning aspect of what we were doing are key to many, many careers. So those are things like breaking a problem down. You're stuck. You can't solve it. How are you going to break it down to a problem that you can solve? Or you've got a bug. It's not working. How are you going to figure out where it is? How are you going to fix it? Perhaps my favorite one, and perhaps this is just a beautiful memory I had of a child with one of those aha moments, is how do you make something more efficient? A physical programming language can't have very many pods. And I think in our current version, we have about 21 pods. So you have to use those really efficiently. That means you have to use loops if you want to do things

again, because you don't have enough pods to do it out in a serial fashion. And I remember a child trying to create the program with Jingle Bells. It was just before Christmas. We were all ready to go off on holiday, and she was determined to solve this before any of us could go home. She mapped it all out, and she said, but I don't have enough pods for the last two words. I said, well, you know, we have solved this, so it must be solvable. So she's sitting there and thinking, and her mom looks at her and goes, jingle bells, jingle bells.

And all of a sudden she goes, oh, I get it, I get it. And she reaches for the loop and puts it in a loop. And I think those are the kind of moments, both as a researcher, which are just beautiful to see when your technologies really help someone move forward, but also the kind of thing that we're trying to get children to get at, which is to really understand that they can do things in multiple ways. Who would ever have thought that jingle bells would give someone an aha moment in technology research? So let's talk a bit about some rather cutting edge, ongoing, inclusive design research you're

involved in, where the goal is to create a deeply personal visual agent. What can you tell us about the direction of this research and what it might bode for the future? I think across all of the major industrial research labs and industrial partners in technology, there's a lot of focus on agents and agents as being a way to augment your world with useful information in the moment. We've been working on visual agents, so visual agents are ones that incorporate computer vision.

And I think one of the interesting challenges that come from working in this space is that there are many, many things that we can perceive in the world. You know, our computer vision is getting better by the month, not even by the year, by the month. From when we started to now, the things that we can do are dramatically different. But that's kind of a problem from a human experience point of view, because what's my agent going to tell me now that I can recognize everything and recognize relationships between things and I can recognize people? Now we have this relevance problem. What am I going to surface and actually tell the person which is relevant to them in their particular context? is how do we make things personalized to people without using either a lot of their data or asking them to do things that require a deep understanding of computer science. So that's a real challenge of how we build new kinds of algorithms and new kinds of interfaces

to work hand in hand with agents to get the experience that people want without having to put too much effort in. So I want to talk about a topic I've discussed with several guests on the podcast. It's this trend toward cross or multidisciplinary research. And I know that's important to you. Tell us how you view this trend, even the need to work across disciplines in the research you're doing today. Well, I can't think of a project I've ever worked on in technology that hasn't required working across disciplines. I think if you really want to impact people, that requires people with lots of different kinds of expertise. When I first started doing research as a PhD, I started right away working with clinicians, with social scientists, with computer scientists. That was a small team at the time. The Torino

project that I've just discussed, we were quite a large team. We had hardware engineers, software engineers, UX designers, user researchers, social scientists involved, industrial designers as well. Everyone needed to bring together their particular perspective to enable that system to be built. And I feel in some ways incredibly privileged to work at Microsoft Research where I sit on a floor with all those people. So it's just a lunch conversation away to get the expertise you need to really think about how can I get this aspect of what I'm trying to solve. You know, there's some interesting and even serious challenges that arise in the area of safety and privacy when we talk about technologies that impact human health. You've alluded to that

earlier. So as we extend our reach, we also extend our risk. Is there anything that keeps you up at night about what you're doing and how are you addressing those challenges? No doubt any technology that uses computer vision sets many people into a worried expression. What are you capturing? What are you doing with it? So I've certainly thought quite a lot and quite deeply about what we do and why we do it. And I think working in the health and disability space has been a really interesting space to work with these technologies because you can see on the one hand that they can have a profound impact on the lives of the people that you're working with. And when I say profound, I don't mean, you know, they had a nicer day. I mean,

they can have lives and careers that they couldn't consider otherwise. That said, we are no doubt with vision technology capturing other people. But for me, that's one of the most exciting design spaces that we can work in. We can start to think about how do we build systems in which users and bystanders have enough feedback that they can make choices in the use of that system? So it used to be that users of the systems were the ones that controlled the system. But I think we're moving into an era where we allow people to participate in systems, even when they're not the direct user of those systems. And I think SSMS was a good example because there we were also capturing clinical data of people. We had to be very careful about balancing the need to, for example, look at that data to figure out where algorithms were going wrong and respecting the privacy of the individuals as there's no way

to anonymize the data. So I can assure you, we thought very hard about how we do that within our team, but it was also a very interesting discussion with some of our colleagues who are working in cloud computing to say, you know, there's a real open challenge here, which hopefully won't be open too much longer, about how we deal with clinical data, how we allow machine learning algorithms to work on data so not everyone can see all of the same data. So it's certainly top of mind in how we do that ethically and respectfully, and of course, legally, now that we have many legal structures in place. Cecily, tell us a bit about yourself. Your undergrad is in anthropology, and then you got a diploma and a PhD in computer science. How and why did that happen? And how did

you end up working in Microsoft Research? Well, I suppose life never takes the direction you quite expect. Certainly hasn't for me. I did a lot of maths and science as a high school student, but I was getting a little bit frustrated because I really liked understanding people. And what I really liked about anthropology was it was a very systematic way of looking at human behavior and how different behaviors could adjust the system in different ways. And that to me was a little bit like some of the math that I was doing, but just with people. So I was solving the same kind of problems, but using people and systems rather than equations. So I found that very interesting.

I went off to do a Fulbright scholarship in Hungary. I was studying the role of traditional music, in particular bagpipe music, in the changes in political regimes in Hungary. And as part of that, I spent a couple of years there. I found some really interesting things with children. I started teaching kids. I started working with them on robotics just because, well, it was fun. And having done that, I was then

seeing that actually there could be a lot of better ways to build technology that supports interaction between children and classrooms. So off I set myself to find a way to build better technologies. I figured I needed to know something about computing first. So I thought I would do a diploma in computer science. But that, again, distracted me when I was given this opportunity to work in the healthcare space. And I realized that really what I wanted to do was create technology that enabled people in ways they wanted to be enabled, whether that be education or health or disability. So I ended up doing a PhD in computing, and then very quickly moving into working in technology in the NHS.

And soon after that, I came to Microsoft to work on the SSMS project. So you have two boys currently, 11 months and six years. Do you feel like kids in general and your specific boys are informing your work? And how has that impacted things as you see them from a research perspective? Again, one of the serendipities of life, you can get frustrated with them or you can take them and run with them. So I have an older child who was born just before I started at Microsoft who is blind. And I have another 11-month-old baby who we call him a classic. We have the new age and the classic version. And it very much

has impacted my work, seeing the world in a different perspective, taking part in communities that I wouldn't otherwise have seen or taken part of, has definitely driven what we've done. So Torino is certainly an example of that. But a lot of the work I've done around inclusive design is driven very much by that. And I think, interestingly enough, in the agent space, we have done some work with people who are blind and low vision because at the time we started working with agents, typical people were not heavy users of agents. In fact, most people thought they were toys. Whereas for people who are blind and low vision, they were early adopters and heavy users of agent technologies and really could work with us to help push the boundaries of what these technologies can do. If you're not using technology regularly, you can't really imagine what the next steps were.

So it's a great example of inclusive design where we can work with this cohort of young, very able, blind people to help us think about what agents of the future are going to look like for all of us. So while we're on the topic of you, you're a successful young woman doing high-tech research. What was your path to getting interested? Was it just natural or did you have role models or inspirations? Who were your influences? Well, I think as maybe some of the stories I've said so far, you can see serendipity has played a substantial role in my life.

And I guess I'm grateful to my parents for being very proactive in helping me accept serendipity and running with it wherever it has taken me. I think I've been very lucky to have a boss and mentor, Abby Sellen, many people may know from the HCI community, who's been amazingly adept at navigating, building great technology and navigating the needs we all have as people in our own personal lives. I'm sure there have been many other people. I take inspiration wherever it's offered. As we close, Cecily, I'd like you to share some personal advice or words of wisdom.

What you're doing is really inspirational and really interesting. How could academically-minded people in any discipline get involved in building technologies that matter to people like you? I think knowing about the world helps you build technologies that matter. To take an example from the blind space, I've seen a lot of technology out there where people build technology because they want to do good, but they don't know how to do good because they don't know the people they're designing for and building. We have lots of techniques for getting to know people. But I think some ways the best is just to go out and have a life outside of your academic world that you can draw inspiration from. Go find people. Go talk to people.

Go volunteer with people. To me, if we want to build technologies that matter to people, we need to spend a good part of our life with people, understanding what matters to them. And that's something that drives me as a person. And I think it then comes into the way I think about technology. Another thing to say is be open to serendipity, be open to the things that cross your path. And I know as academic researchers, sometimes we feel that we need to define ourselves. And perhaps that's important, although it's never been the way that I your path. And I know as academic researchers, sometimes we feel that we need to define ourselves. And perhaps that's important, although it's never been the way that I've worked. But I think there's also something about, you can be incredibly genuine if you go with things that

are really meaningful to you. And being genuine in what you do gives you insights that nobody else will have. I never expected to have a blind child, but I think it's been incredibly impactful in the way I approach my life and the way I approach the technology I build. And I don't think I would have innovated in the same way if I had not had that sort of deep experience of living life in a different way. Cecily Morrison, thanks for joining us today.

Thanks very much. To learn more about Dr. Cecily Morrison, thanks for joining us today. Thanks very much. To learn more about Dr. Cecily Morrison and how researchers are using innovative approaches to empower people to do things they couldn't do before, visit Microsoft.com slash research.