Microsoft Research Podcast - 106 - Microsoft Scheduler and dawn of Intelligent PDAs with Dr. Pamela Bhattacharya

Episode Date: February 12, 2020

In a world where productivity is paramount and only a handful of people have personal assistants, many of us are frustrated by the amount of time we spend in meetings, and worse, the amount time we spend planning, scheduling and rescheduling those meetings! Fortunately, Dr. Pamela Bhattacharya, a Principal Applied Scientist in Microsoft’s Outlook group, wants to turn your email into your own personal assistant. And a smart one at that! Today, Dr. Bhattacharya tells us all about Scheduler, Microsoft’s virtual personal assistant, and how her team is using machine learning to put the “I” in intelligent PDAs. She also talks about how understanding different levels of automation can help us set the right expectations for our experience with AI, and explains how, in the workplace of the future, we might actually achieve more by doing less. https://www.microsoft.com/research  

Doodle did a survey of more than, I believe, 1,500 professionals, and what they found is, on average, people in these roles spend more than five hours in scheduling meetings, just back and forth in trying to find the right time. Imagine five hours every person spending every week to schedule meetings. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Huizinga. In a world where productivity is paramount and only a handful of people have personal assistants, many of us are frustrated by the amount of time we spend in meetings,

and worse, the amount of time we spend planning, scheduling, and rescheduling those meetings. Fortunately, Dr. Pamela Bhattacharya, a principal applied scientist in Microsoft's Outlook group, wants to turn your email into your own personal assistant, and a smart one at that. Today, Dr. Bhattacharya tells us all about Scheduler, Microsoft's virtual personal assistant, and how her team is using machine learning to put the eye in intelligent PDAs. She also talks about how understanding different levels of automation can help us set the right expectations for our experience with AI, and explains how, in the workplace of the future, we might actually achieve more by doing less.

That and much more on this episode of the Microsoft Research Podcast. Pamela Bhattacharya, welcome to the podcast. Thank you. Thank you for having me. That's a great name. Does it mean something? No, I don't think it has a meaning. I bet it does somewhere, but we don't care. Maybe. I don't know. That's your homework for the...

Yeah, I think my sister will be better at that. She did this whole family tree for our entire family. So yeah, I think she might know. I'll get her on the podcast next week. Well, let's start by getting you situated for our audience. You're a principal applied scientist at Microsoft, and you're currently working in the Outlook product group. I'll have you tell us that story a little bit later, but first I want you to tell us more generally about what excites you about the work you do and what gets you up in the morning.

I think I really believe in Microsoft's core mission, which is to empower every individual and every organization in the morning? I think I really believe in Microsoft's core mission, which is to empower every individual and every organization in the world. And I think productivity is the core to it. And I think after working in the space for almost four years now, it seems counterintuitive. But in terms of productivity, doing less is achieving more. Finding that right balance is very important. And how we empower people to find their own balance, because everybody is different, their scenarios are different, it's all contextual. How we help people find that balance for themselves, I think, is what excites me and keeps me going. I want to go in on doing less is achieving more.

Can you unpack it a little bit? Yeah, it was very personal for me. As I was progressing in my career at Microsoft, I was working on different projects. And even within the same team, I was trying to have impact on different things. And then, you know, I had my son. And suddenly the life work balance, you know, what it, changed. But I felt like I still wanted to do impactful work. And I felt like the only way I could continue to do impactful work is by choosing problems that are really, really important and doing only one or two of them rather than doing eight or nine of them, which probably I was doing earlier. And I felt like I became more productive

by making that conscious decision. Almost every day I have to push myself to think like, you know, what can I cut? What can I not do and still achieve what I want to achieve? And I think that's where we can help build tech for people to make them more conscious and aware of that. I want to hover at 10,000 feet for a minute and talk somewhat philosophically about a pretty big topic, and it's the future of work. And it was a theme of MSR's Faculty Summit last year, and the premise, which I quote from the conference overview, is new advances in computing are transforming existing work

and productivity paradigms. Tomorrow, we will work in more places, faster, more collaboratively, and our output will be ever more thoughtful, creative, and impactful. That's a big assertion. From your perspective, and this is just your perspective, but what's hope and what's hype about that statement? I think the hope is that if we can all collaborate in a way that makes us individually productive and globally productive, just to give an example. So on days, I'm maybe just coding and I feel productive at the end of the day because I finished my feature. Right. And on some days I don't even write a line of code. And all I did was brainstorm with other people, you know, help unblock others, you know, whiteboard and try to find solutions. And I still feel productive

at the day because, you know, I did so much, which cannot be really measured in terms of lines of code or, you know, what I delivered, how many presentations, nothing like that. But I feel that one of the major challenges in the productivity space is coming up a very tangible way to measure how productive I was or an organization was. You know, we have productivity scores, but at the end of the day, how do I feel about it? How does my team feel about it? I feel that there is a gap in what we are doing and what we are measuring and how that all ties back as a feedback loop, for example, and how we keep improving. I've been fascinated by the diversity of research interests among the scientists here, and many of them have self-identified. I'm not even kidding, they have. There's a research grid

where you're either like Niels Bohr, who was all about basic research, or you're Thomas Edison, who was all about applied research, or you're Louis Pasteur, who's some mix of applied and basic. But where do you self-identify on that grid? What kinds of projects fall in your wheelhouse? And what value do you think your work brings to the world? I think it's definitely in the applied quadrant. I feel one thing that I really enjoy is problem solving. And it could be that I come up with the most simplest solution that works, or it could be I worked on it, and it's very complicated, and it's very layered, so on and so forth. So for me, the complexity of the solution doesn't change my approach to the problem. If it works, it works. You know, so that's what I care most about.

And we can talk later about, you know, my journey in calendar.help. But after I joined Microsoft, I moved to a team which was just starting. I was literally the first hire on the team. And I didn't think about being in a startup environment or being in that culture or doing something, you know, grounds up.

But that was a grounds up initiative that we were taking at Microsoft to transform customer support experience. And I just loved the experience, you know, having a small team, having much more connection with the leaders and the managers and making sure I really understand what they are trying to achieve and how I am aligned with their goals and so on and so forth. And just the agility of it. That's when I realized that these are the kind of incubation teams that I really find making more impact. I enjoy being in incubation efforts and taking them to completion. Well, let's talk about one specific incubation project that you've just mentioned. It's getting a lot of attention right now. And it was called, as you said, calendar.help.

Yep. But it's now called Scheduler. And you referred to it as both. I'll frame it as a personal assistant for people who don't have people. Before we get into the specifics on the technology, set up the problem for us. What was the pain point that prompted the project? In other words, why do we need it? You might be familiar with Doodle, which is an online scheduling tool.

And Doodle did a survey of more than, I believe, 1,500 professionals. And what they found is, on average, people in these roles spend more than five hours in scheduling meetings, just back and forth in trying to find the right time. Imagine five hours every person spending every week to schedule meetings. And as you know, even VPs at Microsoft, they have their own executive admins that manage their calendars, manage their time. I've dealt with them. So how do we solve this? And the funny part is that before I joined the project, you know, I always thought it was scheduling a meeting. But in reality,

a meeting has a life cycle. You initiate a meeting after multiple back and forth, it gets scheduled. And most often than not, it needs to be rescheduled, right? Or people need to be added or, you know, you name it.

Like, you know, it has a life on its own. Rarely does a meeting get scheduled. It happens and it's done. I can't remember the last time that that happened. Yeah, exactly. Even with our podcast, right? Snow.

Yeah. So I feel like that's the complexity it brings to it. So it's not like once and done. You know, you don't know how many times you have to iterate over it. Okay. So everything you've just said, I bet is going to resonate with everyone in our audience, except the people who have people. And so that pain point is real. I'm just nodding my head. No one can see it, but I'm going, yep, absolutely. So tell us

about your solution, which is now called Scheduler. It's an intelligent personal digital assistant, IPDA, for those TLA fans. And it brings recent advances in machine learning, and that's the intelligence, to the field of PDAs, Personal Digital Assistance. So I'm going to let you run with this because you do a fantastic job of explaining it. But don't be afraid to get technical. Okay. So let me give you a little bit of background. So Scheduler started with the idea of building a gig economy platform for executive assistants. And that's where it started in Fuse Labs. And soon they realized that of all jobs that executive admins need to do, scheduling is the pain point, like we discussed earlier. with being this human and AI hybrid platform to enable scheduling much more easier. So as an user, I can offload it to the intelligent assistant and they take care of it. So the user experience is

really very much like how, say, you see our VPs interact. So say I'm having an email conversation with my VP, Gaurav, and, you know, at some point he might say, hey, OK, let's just meet and chat. You know, it's getting complicated. And then he'll CC his admin, Crystal, and just say, Crystal, can you find us a time? And he might give more constraints. Find us a time next week or find us a time and add somebody else who is not on the thread. Right. And then on what Crystal does, Crystal's job is to understand those constraints and schedule a meeting. Now, let me get to the technical part.

So the assistant or the service gets this email, but this email contains a lot of information that's not relevant to scheduling. Say we talked about the project, we talked about deadlines, we talked about other meetings. But to the assistant, you are very specifically saying about scheduling one meeting. So the first technical problem that we solved was given a document, how do you find out what are the relevant sentences for a task at hand? So here the task at hand is scheduling. So this was a ranking algorithm that we came up with in finding relevant sentences for that. Okay, so that's step one. Step two, once you have found the relevant sentences, then it's about understanding intent. What are you trying to do? Okay, you're trying to schedule one meeting, recurrent meeting, an online meeting, a phone call, a lunch, because all those will define

how you choose the times or what you do with it, right? And it's very contextual. So you might say, let's have a lunch meeting, right? So then the assistant needs to understand that, okay, I can only look between times, say between 11.30 and one, because nobody has lunch either at 10 or at four, right? Versus, so understanding the context, understanding what are the constraints

that apply to this meeting, you know, your intent is the second step. And the third step is now what do I do with this? Right. How do I process it in a way? And what is my final outcome? So, for example, one of the outcomes could be, OK, I say a recruiter is trying to schedule a meeting with a candidate. The assistant doesn't have access to the candidate's calendar. Right. So they have to reach out just like an admin would reach out and say, hey, Pamela is available times X, Y, Z,

you know, does any of this work? And they'll reply again in absolute natural language saying either yes, all that works, or no, it doesn't work, how about something else? Or they might say, I can do the first and the third, right? Or, you know, even more complicated, it becomes like if say we give three options, two on Tuesday and one on Thursday, they just say Tuesday works. So now we have to do

the job of mapping Tuesday with the options. And, you know, so it's a whole other complicated system that's out there. And then again, you know, as I said earlier, meeting has a life cycle. So it might get rescheduled. you might need to cancel it, you might need to add more people, you might need to drop people because not everybody's available, there's no mutual availability, so on and so forth. So that's one layer of natural language understanding, you know, parsing, interpreting, and then there's another layer of preferences, right, personalizing the assistant for you. Because if I say, hypothetically had a human admin, they would know what times I prefer, right? I might not be a morning person at all.

And they know that. They would know that, you know, over time. But if I have a meeting with somebody, say, in MSR India, they wouldn't hesitate to schedule a meeting at 8 a.m. because, you know, that's a more human time to have a meeting, right? For everyone involved. Exactly. So, again, how do you have personalization, but how do you add context to it, right? So that, you know, again, how do you have personalization, but how do you add context to it? Okay.

So that, you know, it, negation, all of that stuff. And I want to talk to you about that. First, tell us what this product does. How does it work? It's very seamless. It's essentially a similar experience to having a human admin where you just CC the admin, which in this case is an email address, kotana.calendar.help, and the service takes over from there. And anything, any new instructions you need to add, you just send an email to your assistant. Okay, so the assistant is actually an intelligent agent. Yep. And the intelligent agent is going to try to do all of the things that a human would. Let's talk a little bit about the problems that are still out there that you're working on.

Yeah, so we're still working on it, right? And it's a mixture. So initially, we look at your calendar and we try to understand what are your meeting behaviors and preferences, right? But then those all change over time. And again, those have layers of context and preferences, right? But then those all change over time. And again, those have, you know, layers of context with it, right? So I might have morning meetings twice a week, but those are only for meeting with people in India with the MSR team there. Or I might have late meetings,

and that might be only for meetings with the Shudzo team, right? And then during the day, I might have one-on-ones clustered, but say other technical meetings might have more space between them because I might need to come from one meeting, you know, just take some time out for myself and then have a next meeting. So how do you balance all these things out? And that's where we are, you know, trying out reinforcement learning to personalize these recommendations based on the context, you know. So that's one area that we are investing in. One of the most interesting areas that you talked to me about was negation and how a machine deals with what a human can understand really easily. Why don't you explain that? Yeah. So the other complicated part of the entity extraction that I talked to

you about earlier is how the same semantics can be expressed in so many different ways. So for example, I might say, hey, I can meet you Gretchen next week, except Thursday, right? That results to exactly the same time range, if you will, versus if I had said, I can meet you next week, I am oof on Thursday. So that's a very implicit way of saying I'm not available on Thursday. So in both these cases, I can meet next week except Thursday and, you know, breaking it into two sentences where you have an implicit way of saying you're unavailable. The concept of negation is something, you know, that is very important for us because those are the times you do not want the meeting to happen. Yeah. And that's something, you know, again, we are collaborating with MSR and we also have our homegrown solutions that we are trying to do better at.

Right, right, right. All right. We're talking about people, right? How do we get bodies in a room? And if it's a phone meeting, that's one thing. But let's talk about other resources for a second, because it isn't just people. it's how many people, it's whether the rooms are available. For example, there might be people from other buildings that need to come. So what about these other resources? How do you deal with that? You bring up a very good point about location intelligence, right? So resource booking is such an important part of meeting scheduling. Imagine having a meeting time, but then not a room, right? I mean, how often do we change the meeting time because a conference room is not available, right? So that's really crucial. So we also support conference room booking, resource booking. So, you know, as an user, you can just say that, hey, book a room for me, or book a room in building 32, or in building 99, right? And the

agent will then extract the intent that you need a room. And then the entities associated with that intent that, oh, it's supposed to happen in building 99. Or if you don't say something, we just go with your, you know, most recently used or preferred rooms. You also can specify to the agent a list of your preferred rooms. Say our VPs, mostly they'll have their own preferred rooms, you know, reserved rooms. So you can also do that version with the agent. And then we try to book your room. And if you're not able to find a room, you know, we just tell you we were not able to. But internally, what we try to do is we try to find a time when a room is available. So the ranking algorithm changes because now it has a

different constraint to it, which that it requires a room. I was just going to ask you about constraints because each layer is a new constraint. And I would hope as a person, I would be, yeah, I get that. And I know they're all here in building 99 and blah, blah, blah. But there's somebody from over in 32 that needs to come over here. Yeah. To add to that travel time, travel time is something that we haven't started supporting yet. But that's, you know, one of our biggest things that we want to invest in. And, you know, we want to solve because the agent should be intelligent enough to say that, oh, yeah, all day you're free from 11 to 12. You have a meeting till 11 in billing 99. You cannot make it at 11 to billing 32, right? So we need to give that travel time for meetings. So today we don't have that intelligence in our system. Well, let's talk a little bit about where you are with this, because that leads in really well to my question about automation.

And you have a wonderful explanation for levels of automation and where Scheduler lives now and where you're heading by tackling the problems we're just talking about. So let me first, you know, for our audience, give a brief about the levels of automation so that everybody is on the same page. So we draw the analogy from automation in cars, right? So in the automobile industry, the Society of Automobile Engineers, they have done a great job in creating these levels, right? Let me go through the levels. Level zero is like, you know, you are in complete control.

There is no intelligence, you know, no assistance, nothing. Level 1 is like most smart cars. So you're still in complete control, but you have some assistance, right? Level 2, level 3 is more. The assistance is increasing. You can get, you know, sensors when you have a blind spot or your wipers turn on automatically when it's raining. Again, you're doing less to do more. And then level four, you're slowly giving away, delegating more and more of the task of

driving to the agent, but you are still in control. You can respond if there is an emergency. And level five, you are just in your backseat and sleeping. Autopilot. Yeah, autopilot. There you go. So in terms of your personal agents, that's where the spectrum is, right? And we are probably at level two right now, right? So I think this framework is applicable to more than scheduler or anything intelligent because I feel it's a great way to set user expectations. So imagine if I went to buy an intelligent car and I didn't know where in the spectrum it falls and I bought a level two car expecting it's a level five car. I'll be disappointed, right? Or angry.

But if I knew that I'm buying a level two car, I know what I can expect. Similarly with digital assistants. So many of our users, you know, in the initial days, they would expect scheduler to do things, which it's probably a level five expectation. So for example, can you reschedule a different meeting on my calendar to accommodate this meeting? We don't support that, right? But they assume as because humans can do that human admins can do that when you share your calendar with them, when you give them access, they can go to your calendar and reschedule meetings, right? That's a level five. We don't support that. So we did sense frustration from people because they were not aligned with what

the technology was, what the product was. So this gives us a great way to align our product, you know, and where we are with the user expectations, because a lot of happiness in using something or doing something comes from having the right expectations. I'm loving your phrase, we don't support that. I think I'm going to start using it instead of I can't do that. I don't support that. Scheduler is literally a poster child for tech transfer stories. Give us a short biography of the project and your association with it. Where was it born?

Where does it live now? And where is it in the pipeline to shipping so I can use it? Yeah. So Scheduler back then started as Calendar.help. It started in the Future of User Social Experiences Lab, also known as the FUSE Labs at Microsoft Research. As I said earlier, it started as a gig economy platform to help executive admins. But then the original team quickly realized that scheduling is a big opportunity in that domain. And that's where it was born. I joined the team in October 2016. And that was when Microsoft acquired Genie, which was a startup in the Bay Area. And this was founded by my current

manager, Charles, and his co-founder, Ben. And Microsoft acquired Genie. And the goal was that we will transfer this over to Microsoft Outlook and it will become a product there. I didn't know where the project was going. You know, I heard about the announcement and I read about Genie and I read about Calendar.help. I didn't know about these projects or these tools or anything. And I just wanted to be a part of it. Yeah. So where is it now in the pipeline? Yeah. So we have been in a public preview program since November of 2016. And last Ignite, November 2019, we announced we are going GA sometime this year. Well, the core technology behind Scheduler seems like it could have other applications.

You've actually kind of alluded to that already. How could you envision using the core technology that's behind this beyond calendars and scheduling? Yeah, I think the core technology is about understanding intent in natural language. And today we only support emails as a written form of natural language, but it can be spoken, right? And also understanding context and preferences. So if I had to layer these, right, the first would be understanding intent. Second is how do you extract the right information knowing the intent of an user? And then how do you find out what their preferences would be in that context and then get the job done, right?

And that can be applied to so many things, reserving restaurants, finding the right times to meet your family, or, you know, when everybody's available, you know, there are myriad of scenarios where you can use that. Well, we've talked about what gets you up in the morning, but this is the part of the podcast where I ask what keeps you up at night. So I think about the work you're doing, helping the assistantless among us get more done with their time, even if we're doing less by achieving more. And that all feels good to me. But I'm sure there are some consequences intended or otherwise that might need to be addressed. So what kinds of things keep you up at night and what are you doing about it?

Yeah. So there are really two areas that makes me anxious about, you know, the progress we are making and what we can do better. So one is definitely, you know, privacy, user privacy and ethics and what I think is currently termed as compliance-constrained AI, right? You have to put the user's privacy as a first-class citizen in everything you are doing. So I think that's definitely a big area where I know many, many, many teams and Microsoft is investing. And I think we have a long way to go there. How can we build even powerful models, but making sure that we are honoring the privacy of the users?

Right. And I think the second bucket is more about user education and how can we bring something intelligent, but so seamless for people to use who might not know anything about tech, right? There are all these complex pieces that we are building together, but how do you make it very easy for people to start using it? And I'll give the context of schedule itself, right? Many times when, say, the organizer of the meeting, you know, they have cc'd Kotana and Kotana had reached out to the invitee saying that, hey, you know, something, I need this information from you. Many times those invite people who receive that email from Kotana do not know how to react to it. Should I write an email? It's an agent, it's a bot, like, you know, what should I do with it, right? So it doesn't scale for us to go one on one and tell everybody, you know,

how you use the product, right? So how can we make that so seamless? Because in the core, the service itself is so complicated, right? But then how do you just almost like flip the coin and the person who is using it or is on the other side of not just the user, but, you know, the invitees of the meeting or anybody involved, you know, in the service, to them, it feels seamless. I feel that is very core. And today I feel that a lot of missing parts where we could do more. Well, it's story time, Pamela, and I would love to hear yours. Tell us a little bit about yourself. What got you started along the path to automating my scheduling problems? And how's your own about Microsoft? So I came to the United States to get my PhD. And along the way, I did an internship at Microsoft

and I got a full time offer. It was more a personal decision at that point to join Microsoft because me and my husband, you know, my boyfriend at the time, we were in a long distance relationship for almost six years and he got an offer from MSR and we felt like, you know, this was a great opportunity for us to be together. And the reason I say that is there was no applied science role eight years back at Microsoft, you know, data, machine learning, AI, these were not, you know, first class citizens that, you know, as a company we were looking at or investing in. So I didn't really know how my muscle that I built at grad school during my PhD would be useful here. But luckily for me, you know, in almost like one and a half years after joining Microsoft, we had this applied science role. And that's, you know, how I started with this team I was talking about earlier as the first person, the first hire on the team where we wanted to use machine learning and AI to transform

customer interactions and customer support in Office 365. So I believe it just worked out for me in that way, in like how I always wanted to be an applied researcher. So that was about how I came to Microsoft and found a role that I really enjoy doing on a day to day basis. And then, you know, about scheduler, I saw the announcement over email, you know, when Microsoft has an accusation, they usually send out an email to the larger org that, hey, we are doing this, we are investing in this. And I found out and, you know, I was very lucky that my VP, Gaurav, who is the VP for Outlook, he introduced me to Charles, who is my current manager. And, you know, because I was so interested in working in this space and things just happened from there. I mean, yeah. So where did you do your doctorate? From University of California, Riverside. So you started in India, undergrad and everything. Where in India were you living?

So I was born and brought up in the suburbs of Calcutta. And my family now moved to Calcutta. But yeah, I was born there. And I think, again, it ties so much back to how much my family believed in STEM education. And my parents, you know, from a very early age, they made sure that, you know, education is important, math is important. And actually, my dad, I believe, was more interested for me to pursue computer science. Both my parents, actually. Then I don't think I knew what it meant, you know, when I signed up for coding classes. But I enjoyed it, you know, and I think that's how things went. What's one thing people might not know about you, whether it's a personal characteristic or a life-defining moment or a personality trait or a side quest that may have influenced your decision to become an applied scientist?

So I think I did not start thinking I had a lot of role models, right, like people I wanted to be like, and I wanted to make an impact the way they were making. Right. And I think that all these people that I really but they have put in a lot of effort to get there. So that was one thing. And the second thing is that if I role model someone, I don't have to be exactly what they are doing. I have to kind of extract out what in them do I really appreciate and how can I build that muscle? So I felt like it was more about what are they doing and how are they doing and how can I apply that to my scenario and take it from there? So I think that internalizing that helped me in my path forward. It sounds like you're pretty analytical about things. It's funny you say that because my manager told me the same thing in my last Connects.

Because we were talking about a very different problem, like not even with the product. And he said, you know, Pamela, you're very analytical. Yeah, I guess I am. Yeah. As we close, and I'm sad to close because it's so delightful talking to you. Yeah, same here. Thank you. I want to give you the opportunity to talk to your grad school self, the one who didn't know where her career path would take her

and would love some advice from her future self. So what would you say to you and by extension to those in our audience who are where you were then right now? I think perseverance is really important. And sometimes we don't realize even people who we know as really successful, how much failures they have gone through. And I feel that is something that I would tell my younger self is just coming back every day and trying to set the right expectations from yourself, being kind to yourself and whatever your dreams are. You should go for them and you should just keep doing it. And not give up. Yeah, not give up. It reminds me of a phrase I've heard that I really like.

Success isn't permanent. Failure isn't terminal. Absolutely. Yeah, yeah. That's great. That kind of summarizes. Pamela Bhattacharya, thank you so much for joining us today. It's been an absolute delight. Thank you. It's an honor. Thank you for having me on your show. To learn more about Dr. Pamela Bhattacharya and how researchers are working to make life easier for people who don't have people, visit microsoft.com slash research.