Microsoft Research Podcast - 128 - New Future of Work: How developer collaboration and productivity are changing in a hybrid work model

Episode Date: July 14, 2021

For Microsoft researchers, COVID-19 was a call to action. The reimagining of work practices had long been an area of study, but existing and new questions that needed immediate answers surfaced as companies and their employees quickly adjusted to significantly different working conditions. Teams from across the Microsoft organizational chart pooled their unique expertise together under The New Future of Work initiative. The results have informed product features designed to better support remote work and are now being used to help companies, including Microsoft, usher their workforces into a future of hybrid work. In this episode of The New Future of Work series, Chief Scientist Jaime Teevan and Principal Productivity Engineer Brian Houck discuss what the massive shift to remote work meant for developers—both employees of Microsoft and customers using Microsoft developer platforms to support their work. They’ll talk about how taking a holistic approach to developer productivity can benefit both efficiency and happiness, with an emphasis on the important role social connections and processes play in a field often thought of as an isolated endeavor. They also explore pros and cons of everyday developer tasks, like code review and whiteboarding, being done in a hybrid work setting. https://www.microsoft.com/research

Some of the classic measures of developer productivity really are looking at output. And so things like number of pull requests completed or number of features shipped or even bugs fixed. And what we found during the shift to remote work is that's not a great measure of productivity because it doesn't speak to the cost, you know, the human element of how to create those. And so we really want to take a broader look where we look at, well, what are some of the customer outcomes? Like, what was the cost of those activity metrics? You know, how happy are our developers and how efficiently were they able to do their jobs?

And so when we look at measuring developer productivity, one of the things we've learned is there is no one measure. Welcome to the Microsoft Research Podcast, where you get a front row seat to conversations on cutting edge technology. I'm Jamie T. Vann, and I'll be your host as we investigate how work practices have changed because of COVID-19 and what it means for creating a new and better future of work. Thus far, we've talked about the impact COVID-19 has had on how information workers in general

get things done. Now we're going to turn to a specific subpopulation of information workers that's very important to Microsoft and technology companies, and actually increasingly to all companies, namely software engineers. Brian Houck is joining us to discuss the chapter that he authored with Chandra Madilla and Peggy Story on software engineering experiences in the new future of work report. Brian leads the Azure Developer Productivity Program Management team here at Microsoft. That team focuses on increasing the productivity of Azure and Windows developers. Today, he'll be sharing with us some findings from the research that he and his teammates have done this past year, as well as synthesizing the research coming in from across Microsoft and the external community related to how COVID has impacted software development.

Welcome, Brian. Thanks so much for having me, Jamie. Oh, thank you. So software engineering experiences are called out in particular in the report. Can you say a little bit about why we did that? Why is Microsoft interested in software engineering experiences? You know, it's interesting from a couple of different perspectives. Obviously, Microsoft is one of the largest software development companies in the world. So we employ a lot of software developers and we want to make sure that some of the changes around us, that we still are able to meet the needs of those customers who are developing on our various platforms. And one of the things that I found fascinating is how rich a picture Microsoft has into development practices. Can you speak a little to the different developer communities

that Microsoft touches and the ways that we can study their work? Yeah, absolutely. Microsoft is uniquely positioned in the industry to sort of understand the changing experiences for developers because we touch such a breadth of these communities. You know, through Windows, we build the platform that many developers are using in their day-to-day jobs. And there are a variety of communities of passionate developers who are members of our Windows Insider community, as an example, as our MVPs. Microsoft also builds many of the development tools that external developers are using through Visual Studio and Visual Studio Code, who also have robust communities of passionate users. And then lastly, you know, Microsoft has recently acquired

GitHub, which is more than anything, a community building space for developers to come together and collaborate and co-create on software projects. And that has rich community features. And so from the entire stack of the platform, the tools, and then the end work product, there are tools that Microsoft are building, and then the communities that use them that we're able to interact with and engage with on a regular basis. You know, earlier, Sonia and I were talking a little bit about some of the challenges related to measuring productivity. And I'm curious, what are some of the different ways that we can measure developer productivity?

Oh, the age-old question of how to measure developer productivity. One of the things that I have learned as I have been looking in this space is there is no silver bullet. There is no one measure of developer productivity. It is multiple dimensions that come together and perspectives matter. Like as an IC developer, I may have a different view of productivity than what my manager may have. And some of the, you know, classic measures of developer productivity really are looking at output. And so things like number of pull requests completed or number of, you know, features shipped or even bugs fixed. And what we found during the shift to remote work is that's not a great measure of productivity

because it doesn't speak to the cost, the human element of how to create those. And so we really want to take a broader look where we look at, well, what are some of the customer outcomes? Like, what was the cost of those activity metrics? You know, how happy are our developers and how efficiently were they able to do their jobs? And so when we look at measuring developer productivity, one of the things we've learned is there is no one measure. It's how can we take a breadth of measures and sort of triangulate them together in order to tell a complete story about productivity of developers. So take a couple of these different metrics that you were talking about, whether it's pull requests or, you know, self-reported experience, and talk me through what sort of changes we saw in those measures from before COVID and after COVID? Yeah. So one of the things that I was really interested in very early is just binary. Could

developers do their jobs? Like, where are tools going to stand up to this very sudden shift to remote work? And that's a place where activity metrics are useful. Like, they can tell us about the health of our systems. And what we did see is that not only was it possible to do the jobs, but we saw through these activity metrics a very large spike in activity very, very quickly. And so we saw, particularly in the early days of the shift to mandatory work from home, that our internal developers were producing a lot more. We saw pull requests go up. We saw total number of hours work going up based on when pull requests were being created. And across all activity metrics,

we saw more features being completed, more code reviews being participated in. There was just more activity coming from our developers, particularly in those early days. And what we found is that even though we were doing more activity, I wouldn't necessarily say that we were being more productive. And this sort of goes to this difference between output and productivity, where we were creating more things, but it was coming at a high human cost. We were throwing more hours at it, and it was leading to some overall well-being challenges around feeling

burned out and feeling stressed. Yeah, so developers seem to have experienced a similar hit to their well-being that information workers did in general. Yeah, absolutely. And oftentimes development is thought of more as an isolated activity. Developer can sit in front of their computer and write their code, but that's not the reality anymore. Like development very much is a team sport. And as we were cut off from our teammates, certainly we saw the same well-being concerns in developers that we saw across workers in every industry. You know, it often seems like some of the things that are good about remote work are the same things that make it really crappy. You know, like to what you were just saying, like focus time is awesome, but then we miss those serendipitous interruptions or

the connections with people. Do you have any insight into what's going on there? Yes. One of the things that we looked at early is what were the top challenges and the top benefits that our developers were experiencing. And we really did see that they were two sides of the same coin. People were enjoying more flexibility in how they could do their work, but they were experiencing a challenge of lack of work-life boundary. And so while we enjoyed that flexibility, we weren't actually able to apply it in a way that was healthy. And we saw that individual experiences vary dramatically. We saw that some people felt they had less distractions during the day, And some people felt that they had more distractions during the day. But what was even more interesting to me is that some people felt they both had less distractions and more distractions. And so like their daily experiences vary dramatically.

And so while sometimes people could focus more because they had less people interrupting them, they had other life events interrupting them. And so it was this very interesting, what we called a tale of two cities. It was the best of times, it was the worst of times. And there was this very close relationship to the challenges and benefits that we were experiencing. Yeah, I love that point you make too

about the sort of variation, even just being within an individual. Like some days are awesome and some days are really hard i found the study of well-being days super interesting and impactful since it's part of why Microsoft is now offering a number of well-being days this year. Can you tell us a little bit about what well-being days are and what the research tells us about their impact? Yeah, I really thought this was some of the most important work I have done over the last year. What we had noticed is that the natural rhythms of taking vacation days just weren't occurring during the pandemic. Because, you know, when society is sort of closed around us,

what are you going to go do? Like, there are no other things you could do. You're just sort of sitting at home. And so people weren't taking vacation days because in many ways work was a welcome distraction. But unfortunately, that was leading to themes of burnout. We heard that people were feeling overworked. We know that nearly 80% of all developers at Microsoft felt like they were experiencing a lack of work-life boundary. And as you went and talked to developers in one know, in one-on-one interviews, it was something that came up in almost every single one is this notion of burnout.

And so what we were interested in looking at is, can we interrupt those themes of burnout and take some shared days off to invest in our emotional and physical well-being? And so we ran an experiment where in an org of about 4,000 developers, we said, okay, we're all going to take a shared pause for two days, a Friday and the following Monday to have a four-day weekend. And what we found is over that four-day span that encompassed the health days, we did see activity dramatically fall off. It was about a third of what we saw over the same corresponding four-day weekend in 2019. And so that was an indication that developers really were taking the pause. We did actually step back and take that break. But what we saw was that we made up for all of those lost pull requests within just two weeks coming out of those shared wellness days.

And so if you had zoomed out and looked over two weeks, you wouldn't have known that the org took any days off. And so there wasn't any interruption in overall output or activity coming out of the org. And so, you know, if you just looked at the data, you might think that, oh, well, all of the developers just went back to their frantic pace and immediately got burned out again. And that wasn't actually true. So I was doing a set of rotating one-on-one dev interviews with two to three devs every week. And prior to those shared health days, you know, feelings of burnout came out in every one of the interviews or nearly every one of the interviews.

And then after those shared health days, burnout wasn't a theme mentioned for the preceding 14 weeks. And so they actually provided some meaningful relief to the org, even though they didn't come at the cost of any output. And I think that just goes to show that investing in emotional well-being actually helps support sustained productivity. It doesn't come at the cost of it. And it's a great example of how to take a rich perspective of what people are doing by looking both at their pull requests and people's feedback. The pull requests are super interesting because I think some sort of metric of what people are doing is useful, even if it's just a proxy. There's another study I found really interesting, which was the work that was done related to onboarding. Can you tell us a little bit about what we learned there? Yeah. So I was really interested to see, are there specific cohorts

who seem to be doing better or worse during the shift to remote work? And what we found when looking broadly across software engineers is that as we got more used to working remotely, we were enjoying it more. And our productivity was in fact going up and many people were excited about the prospect of working remotely at least part of the time after the pandemic ended.

But we did see that new hires were struggling. And the first warning sign we saw was looking at the time to first PR for new hires and how many pull requests they complete within their first 90 days. Looking at new hires who were joining during this period of remote work compared to new hires who had joined during the same time period of 2019. And what we did see is that PRs per new hire had dropped 23%. And it's not just that the PRs dropped. We actually were able to see that only a third of our software engineering new hires reported feeling socially connected to the team.

And so there was this problem of building a sense of connectedness, building up social capital for these new hires who never had the chance to meet with people face to face. And so where more tenured software engineers like myself might have been able to spend down the social capital that I built up over a decade and a half at Microsoft, these new hires didn't have any of that social capital. And so even when we compared them to new hires who joined immediately before work from home, we saw more troubling outcomes. So even those new hires who had just a small amount of time to meet with their team face-to-face seemed to be building more connections. We could actually see that they had

broader and deeper networks of people they were talking to, and they were having higher productivity outcomes as measured by pull requests. And so this is a place that we really need to figure out how to effectively onboard new hires and help them build up social capital, even in a remote world. What are some of the things that software engineers seem to have missed most about working in the office? It's really interesting. It varies so dramatically person to person based on certain experiences. One thing we hear a lot about is whiteboarding. There is sort of this cultural tradition of you get a bunch of developers in a room around a whiteboard and you brainstorm these tough problems and you write out your architectural diagrams. And many, many developers tell me that the number one thing they miss is getting in a room with their team and whiteboarding. Now, on the flip side, there are plenty of people who have also told me they prefer digital whiteboarding solutions because they can be more inclusive.

And when you're whiteboarding on a physical whiteboard, someone has the marker and is physically in front of the whiteboard, sort of blocking access to it. But with digital whiteboarding solutions, anyone who has a pen can participate. It sort of democratizes whiteboarding. And so while whiteboarding was a consistent theme, I heard, it's something that developers were missing. There are other developers who actually felt that this was an opportunity to build some more inclusive, collaborative processes. We also heard just like any other humans, we missed social interactions.

We were able to see that the majority of developers at Microsoft felt it was difficult to communicate with colleagues during COVID, and we were able to measure that those who felt it was difficult to communicate with colleagues during COVID. And we were able to measure that those who felt it was difficult to communicate with colleagues also reported being 13% less productive. And so we missed these human interactions. And we can measure that actually impacted our productivity.

And so it's not just dev-specific things like whiteboarding, but it's these human moments that everyone is missing. And devs are certainly not immune to that as well. Do you think software engineering is going to go back to the way it used to look, or is it going to look a lot different? I think it's going to look a lot different. What we have found is the vast majority of software engineers plan to work remotely at least part-time moving forward. And so what I think is that the way that we interact with our teams is going to change. I think while you will certainly have people go into the office every day, you will have,

I think, a larger number of people who are permanently remote. I think certainly in the foreseeable time horizon, we're going to be in this hybrid world where people are doing sort of a mix. And it's not just going to be, okay, I'm going to do a set of activities and maybe I'm in the office and maybe I'm not home. I think people are going to restructure when they come into the office versus when they stay at home. And so they may come into the office on times that they want to be a little more collaborative. They want to have some human face-to-face interactions and they'll be at home when they're working on some of their more isolated inner loop coding projects.

And so I think we're going to be in this hybrid world and we're going to have to adapt everything about software development to take that into account. Because on a given day, I might not be in the office at the time with all of my peers. And so if I'm going to whiteboard, I'm going to have to use a digital whiteboarding solution. If I'm going to have a meeting,

well, I might be face-to-face with some of my team, but not the entire team. And so I'm going to have to look at how I operate in a hybrid meeting practice. If I want a code review, I can't necessarily just go and tap the dev next to me on their shoulder. I'm going to have to go and do a virtual code review,

which is fine, but it's going to just change the rhythms of how devs do their work today. And what are you most worried about looking forward? And what are things we can do to kind of get ahead of that? Yeah. So I will say that the thing I am most worried about is giving up some of the gains in inclusivity that we have had over the shift to remote work once we move into a hybrid environment. I think many of our meeting practices today are more inclusive because through tools like Teams, people can raise their hand. They can participate in multiple ways, like via the side chat versus having to talk. And so it's a lot easier for people to participate in some of these human interactions than maybe we did face-to-face,

where sometimes the loudest voice is the one that gets all of the airtime. And my fear is, as we move to a hybrid world where some people are remote and some people are in the office, it's going to be easy to fall back into old patterns where it's the loudest person in the room is the one who gets to talk versus can we actually have moderated, healthy discussions where everyone can have their voice heard? And that's the thing that I think we have to keep our eye on the most, followed by effectively onboarding new hires. How do we make sure that we give our newest employees a great start to their career

and make sure they're set up for long-term success and their general wellbeing? We are social creatures and it's making sure that we can help those new hires build a healthy and sustainable network professionally, even when some of their colleagues and peers might be remote and some may be in the office. This is fabulous research that you've been sharing.

And I'm curious, what got you interested in and excited about understanding developer productivity? Yeah, so my day job is building developer tools. And so I work in our internal engineering system. And so I build the tools that our internal Microsoft developers use in order to do their day-to-day code creation. And as part of that, I wanted to be able to measure and articulate the success of the products that I was building. And so I got interested in, well, what are different techniques to measure developer productivity? And from that, I got interested in, okay, well, what are all of the other factors

that influence it? And what I learned is, well, the tools that developers use certainly are important. They're only a small piece of the puzzle. Our environment, our organizational structures, our team processes, and our team culture also matter and impact the productivity and overall well-being of our developers. And so I got really interested in looking at what are all of the things that are providing both negative and positive pressure to the productivity of our developers. And it's, you know, things like the shift to daylight savings time. We're in Seattle. We're pretty far north. The shift to daylight savings time matters. And I can see. Does it matter in a good way or not? Should we get rid of daylight savings or not? Without a doubt, we should always be on daylight savings. We're impacted by the world around us. We can measure how the complications of the most recent U.S. election impacted

productivity. We can look at how open offices versus private offices impact productivity. And so I was interested in all of these factors, again, as a way to look at what were all the things that impacted my day job of building developer tools. And it's through that then when this massive event of COVID happened that I applied some of those lenses to seeing how the shift to remote work impacted productivity. COVID changed developer work practices, but it also must have made it a lot harder to study developer work practices. Were there any particular challenges in doing the research that you do this past year? You know, I actually did not find that to be true. In fact, I found the opposite to be true. So, you know, part of my work is looking at multiple methods. There are

surveys that have been sent around in order to measure developer sentiment. We have a wealth of telemetry at our disposal because at Microsoft, we're building the platform, we're building the tools, and we're building the source repositories. And so we have this really holistic view of the development cycle. And those were at our disposal as much as they ever were. But what I really found is that there was a much greater appetite to talk one-on-one than there was before. lot of willingness to share their experiences in order to, you know, make sure that their perspective, their voices were heard and hopefully help find some good shared solutions for our internal development community. And so I found that it was really easy to solicit volunteers to go and do one-on-one sort of structured guided interviews because people were craving interaction for one,

but more importantly, they just thought if they could share their experiences, maybe we could find some great shared solutions like the wellness days. And so in many ways, it made the research a lot easier. Is there anything else as we wrap up that I forgot to ask you about? The thing that surprised me that I'll just leave with is I got into looking at the shift to remote work through COVID because I wanted to see how our tools were holding up. And turns out, it immediately became clear that our development processes, our tools held up fine. What needed more work, what didn't hold up at the beginning as easily were our social processes. And we see that employees who felt that they were able to

maintain connections with their team, they were able to maintain social connections, had the most positive outcomes. And teams that invested in sort of having this culture of connectedness had the most positive productivity and just general satisfaction outcomes, just like any other knowledge worker. And we can empirically prove it, that investing and maintaining a sense of connectedness is worth it. It's worth the investment. Thank you, Brian. And thanks to our listeners for tuning in.

We hope you'll continue to join us as we explore the new future of work. You can learn a lot more about the research that we discussed today at aka.ms slash new future of work. Also, be sure to subscribe for new episodes wherever you listen to your favorite shows.