Microsoft Research Podcast - What’s Your Story: Nicole Forsgren

Episode Date: February 15, 2024

Partner Research Manager and developer experience expert Nicole Forsgren talks about the future of software engineering with AI, why she loves tech, and her reliance on a spreadsheet and her gut when making career-changing decisions.Learn more:Nicole Forsgren at Microsoft ResearchNicole Forsgren websiteQuantifying the impact of developer experience | Microsoft Azure Blog, January 2024Yes, good DevEx increases productivity. Here is the data. | GitHub blog, January 2024Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations | Book, 2018

. Assume that something can be figured out and that it's not hard. I didn't find out until the end of college that computers were hard. So if it's hard, that's okay. It might mean that you should just spin it on its head and try to take another look. . Microsoft Research works at the cutting edge. But how much do we know about the people behind the science and technology that we create?

This is What's Your Story and I'm Johannes Gerke. In my 10 years with Microsoft, cross-product and research, I've been continuously excited and inspired by the people I work with and I'm curious about how they became the talented and passionate people they are today. So I sat down with some of them. Now I'm sharing their stories with you. In this podcast series, you'll hear from them about how they grew up, the critical choices that shaped their lives, and their advice to others looking to carve a similar path. In this episode, I'm talking with partner research manager Nicole Forsgren.

A leading expert in the field of DevOps and the lead of Developer Experience Lab, Nicole oversees Microsoft research efforts to better understand and enhance the developer experience through the study of their productivity, community, and well-being. Prior to joining Microsoft, Nicole was a successful software engineer at IBM, a college professor, and a co-founder and CEO of a startup that was later acquired by Google. Here's my conversation with Nicole, beginning with her childhood in Idaho. We've talked a little bit beforehand, but you had this amazing career in tech. Tell us a little bit about how you grew up and how did you end up in tech? Yeah, it's kind of this ridiculous story. I grew up in a little farm town

and ended up going to college and thought I would just be there a year or two. A little farm town in? In Idaho. In Idaho. Yeah, so across the street from a potato field. My grandpa was a potato farmer. And where I'm from, girls, a lot of girls go to college, but not usually for very long. You kind of go to school to get married. And so I was majoring in psych and family science. So in high school, you didn't know anything about, or you weren't excited about tech yet or anything? I don't think I knew much about it. Okay. Right? So, I mean, we had a computer at home. My dad was a civil engineer. But I had only ever used it to write papers. I knew Word, Perfect, because back then Word Perfect was kind of the thing.

So it was a PC? It was a PC. It had, you know, reveal codes. Okay. And it was DOS, just not Windows yet, right? Or was it Windows? I which it was DOS just not Windows yet right I wasn't Windows I think it was DOS back then yeah and that was the faces and everything yeah we had all the interfaces and we had a typing class in high school but that that was it and then I went on to college, and a couple of months into my freshman year, my dad, who I was really close with, so I grew up as a tomboy, I was always mountain biking up in the mountains, I was always just like dirty and kind of just playing around.

I was very much a tomboy. Two days after Thanksgiving, he was in a snowmobile accident and in a coma. And suddenly, I sat back and I realized that, you know, all the things that I thought I was going to be able to rely on, like, you know, if I ever needed to fall back on my family or my dad or pay for things, for some reason, like, that was what stood out to me. For some reason, you know, I was in college on a volleyball scholarship, and I, like, I was paying for things, and it was okay. But for some reason, that was what stood out to me, because I was the oldest of four children. The youngest was 11 years younger than me. So I was like, if anything

happens, I think I just felt that responsibility, was that I was going to need to have money. And one of the girls on the practice team, I think in just a totally random comment, said that she could make money in her degree. Because you hadn't chosen that as the... Yeah, it just sort of came, well, because when we went to college, it was, I was going to get married and so then I wouldn't need money, right? Because I didn't choose a degree for that. I want to say it was within a couple of days of my dad's accident, so he was still in a coma. He was still hurt.

And it was a very side comment she had made, and I remember hearing it kind of in the back. And I kind of perked up, and I was like, what do you mean you can make money with your degree? She said, oh yeah, when I graduate with a two-year degree, I can make $40,000 a year. So she wasn't that year, well, she had a university or she was at a different? So I went to a JC first because we don't really go to college. I was like, oh, $40,000. And I remember thinking, well, if I stay in psych and family science, to be able to make that kind of money, I would need to be a high school teacher, which requires a master's degree. And so I talked to her just a little bit. I said, well, what's your degree?

She said, computer information systems. information systems I said well do you like it she said yeah it's really cool it's computers I said oh and what can you make again she said about 40,000 I was like cool and I just remember looking up that degree and going into finding you know I just found the counselors in that degree and telling them I wanted to change my major. Oh, wow. That's a huge decision. I mean, for some reason, did your father recover? So he ended up coming out of his coma within a couple of months, but he was brain damaged for the next few years. And he passed just after I graduated from college.

I'm sorry to hear that. But then, so you went ahead and changed your major and went into? I changed my major without ever having a computer class to CIS, which is now I know, right, computers and business. That summer, I applied for an internship and got it. After your freshman year? After my freshman year. Wow. What kind of internship was it? It was in mainframes for computer hospital systems. And so I showed up and they thought I was

full-time, which I was not. I just interviewed well apparently. And so they threw me the manual, the documentation. What kind of mainframe was it, do you remember? Yeah, it was AS400s, and it was programming RPG and CLI. And it was for… Beautiful languages. They were, I loved it. And it was Siemens, SMS Med Series 4 was the company, later acquired by Siemens. And it was a wonderful team.

And I spent the first week reading the documentation and proofreading the documentation. And then I just kind of dove in. So proofreading, you mean you discovered a bunch of bugs? There were a lot of typos, a couple of bugs, right? Because I just kind of had to figure it out as I went. Wow, so it's not like somebody did pair programming with you at the beginning, you were just handed the manual and then you went with it? I was just handed the manual and then I went with it. Wow, it's an not like somebody did pair programming with you at the beginning, you were just handed the manual, and then you went with it?

I was just handed the manual, and then I went with it. Wow, it's an amazing self-starter. I had to figure out one or two things, and then I just dove in. And then I went back that next year and had my first RPG class. Wow. They were teaching an RPG in college. Yeah, so this was my freshman year was 97, 98, and then my sophomore year was 98, 99. So it was a JC, and they were kind of gearing up one or two classes in anticipation of the Y2K bug.

Right. And so mainframes and financial and hospital systems. Everybody thought the world was ending, or at least some people were thinking the world was ending. They were so worried about it. And so they had brought back one or two mainframe languages to help with the Y2K crisis. And so that summer I did hospital systems. The next summer I did, it was a company called Assistant Kayenta,

and it was like order, it was financial systems and ordering systems. And so I started my career in like, it was a version of mainframes. And then did you get that high paying job afterwards? Yeah, so I had a full time offer, but by then I decided to go for my four year degree. Okay. Transferred to Utah State, which was within a few hours from home. I still had to kind of stay closer to home because of family, and I wanted to, you know, make sure I was close for my dad. But then I did a six-month

internship at IBM. I was a software engineer, and then they hired me full-time when, again, I was a software engineer. But you basically finished the two-year college, then went to Utah State, finished a four-year degree. Finished a four-year degree. Again, there I was, they called it business information systems, now it's management information systems at Utah State. It was a very technical degree, so I was doing network engineering, I was doing databases, I was doing C++ on my programming classes. We're all in the computer science department.

When I got hired at IBM, they hired me as a software engineer. I was working on large-scale enterprise storage systems. Which language did you program in then? There I was doing some C++. I was doing a little bit of C. I was doing some Java, and I was doing a little bit of their firmware. Wow. And then I eventually ended up managing some of their systems, so I did some Bash.

Okay. And then eventually I even got a hardware patent. Wow. What is the patent about? It's kind of everything. It's a way to kind of further obfuscate for cold boot attacks. This really kind of fantastic article came out showing that if you take compressed air and turn the can upside down,

you can freeze some of the bits in chips. Right. And then if you rip the chip out, you can read what's on the chip. Oh, wow. Because it freezes all the electrons on the chip. And we were like, well, this is ridiculous because it's only frozen for two to five seconds. But then if you rip it out and you drop it in like liquid nitrogen, you can read it for two to five minutes.

Right. We're like, well, again, this is ridiculous. But if you've entered in a password, then it's stored in plain text. Right. Well, it's not ridiculous if you get your way into a lab, and at the time I was working in a large lab, because you've entered in the password for one computer, one of the servers, one of the storage servers. And you've destroyed the, you've broken the disk for that one server,

but the password is gonna be the same for every single server in the lab. And so we realized that this is a serious problem. That's a threat vector for a lab. And it's pretty easy to get into labs, right? You can follow anybody in. And so we wrote a patent that kind of further obfuscates and it hides where passwords are stored through malloc calls. I see. So the location of the password was then somehow obfuscated, not so easy to find. Well, so the location of passwords but also additional plaintext strings and other strings are obfuscated throughout the pieces of different areas of chips.

This was motivated by the hardware but then implemented in software? Yeah. Okay. Wow, super interesting. Part of hardware and part of software. Super. I mean, just imagine this career. I guess in high school you were playing volleyball?

In high school and college I was playing volleyball. Ended up getting hurt, so I didn't play volleyball longer. But then switched to MIS. Yeah, switched to MIS. Patent. Patent and shout out to my co-authors on the patent, Ben Doney, Andre Koster. They were great because we all just kind of ended up brainstorming one day. And it was this kind of windy path. But it was, I think it was interesting because at the time, you know, I originally went into tech because I thought I would really need money, but ended up falling in love with it. Right. I had, I've had a lot of fun along the way. So what did you feel in love with tech? What really, what really gets you going on tech?

I love the fact that there are hard, interesting problems that you can solve lots of different ways. And if I can't solve it initially and if I can't solve it the first time, I can just kind of spin it around or pivot it different ways and then just solve it again. So it's this notion that you have this not only hard problems, because in math you have lots of hard problems as well, but here is the experimentation with it and the trial and error. The experimentation and the fact that it's applied, the fact that I can build something and watch it work. Math I liked and math up to a point. I was pretty good at math and I could kind of see how the equations were supposed to work.

Computers and programs helped because I could really see how it was working. Yeah, I think that, I mean, I can so much relate to this because that's how I fell in love with computing. Because you have this machine here and it basically can do everything, right? And I mean, we'll get later to AI, but we see with the current AI systems, right? You can see that, you know, it can be nearly intelligent or it is intelligent, right? And I mean, we'll get later to AI, but we see with the current AI systems, right, you can see that, you know, it can be nearly intelligent or it is intelligent, right? And so it's just amazing to have that kind of machine below you or with you and to help you and to be able to train it and program it. And so you were at IBM and now you're at MSR.

And now I'm at MSR. So what's the bridge in between? Oh, so also kind of a windy path, but it's interesting because as I look back, I guess it makes sense to me. So then I had an opportunity to go get a PhD and I actually started doing kind of large data like NL, you know, natural language. You know, how do we want to think about, you know, analyzing sentiment analysis, analyzing, you know, those types of, you know, big questions like when are people lying? When are people not lying?

Machine learning problems. But I was also working at IBM at the time. Because I continued to like working on systems and working on large problems. So I was doing both at the same time. And I ended up doing a usability study completely randomly. What was the study on? What did you study? So we did a usability study for sysadmins.

Okay. And it was interesting because at the time IBM was trying to build like a GUI for large-scale sysadmins. And so Todd Eichried and I did the usability study. We wrote up the findings for IBM. We shared them. And, you know, we found that in some cases they would use this front-end user interface, and it was fine. And in some cases they were like, I don't know, right? Like, I could click this button, but it really makes me nervous.

And we're like, it's okay. It's a sandbox. This is fake. You can click the button. I'm curious to see what the next step is. And they were like, it's just so risky. And it struck me as being sort of interesting because there were just cases where risk and complexity really interfered with our ability to trust a system or to trust a GUI. And so we wrote this up and we shared it. And IBM was like, you know, we used user-centered guidelines. We used, you know, user-centered design guidelines.

And we were like, but the same guidelines can't work for complex systems and complex distributed systems that can work for laptops. Because in one way I affect this one machine here, but in the other way I affect this row of machines and I don't know how many there actually is. Right, and they really wanted to see the command line interface and the back end data and they really wanted to verify. And so you really had this difference between not just risk and

complexity but also expertise. There were some cases where you could hide complexity and other cases where it just wasn't appropriate, right? And simultaneously I'm working on these really, complexity in other cases where it just wasn't appropriate, right? And, you know, simultaneously, I'm working on these really, really large projects with IBM, and people are just kind of burning out. And I thought, you know, there has to be something to this, right? And there were kind of rumors as I was going to tech conferences still of, you know, this newfangled way to make software and kind of reduce burden. And so I just pivoted. I changed my research project. as I was going to tech conferences still of this new fangled way to make software and kind of reduce burden.

And so I just pivoted. I changed my research project to start studying what now we know of as DevOps. Maybe we'll get to this a little bit, but I'm so impressed by, you know, you go to college, do A, something happens. You do B and don't look back. Same thing here, right? You're a successful developer at IBM. You have this

event which says, wow, I should study this intensively, and you go and get a PhD, right? I mean, it's just super impressive. How do you do that? I will say it's not always super straightforward, but there are times when I kind of sit and I'm like, I'm getting one or two signals. I really want to take a half step back and say, here's an opportunity. Should I take the opportunity or should I not? There have been one or two times where I'm not sure, you know, but there have been times where I'm like, I need to jump. And I can reevaluate in six months or a year, but I'm going to take this now. And I will say about 90% of the time

I have been absolutely on. I find this so fascinating because we all are faced with different opportunities and chances in our lives. How do you evaluate this? Do you have like a checklist? Do you go back and ruminate and meditate on the mountain? Or what's your method? I actually have a spreadsheet. Okay. But I don't always follow it.

So what I like to do is identify like what are what are my criteria? Which is, you know, what things are important to me? Yeah. You know, some of my big moves. What's important to me in the city? Yeah. And then for each of those criteria, or when I am considering a new job, what things are important? And then how important are they? And you put, like, a risk score behind it or like a

score? A risk score or either a risk score or an importance score. And then I'll just kind of multiply it out. And now, then I will go back and I'll like nudge all the numbers. And then sometimes I'll still change my mind because there's something about your gut that says, I don't know. But by identifying all those things first, it helps me think through it. Now, the spreadsheet, like I've shared with folks, and it's really interesting because just the exercise of saying what things are important to me for this decision and how important are they, sometimes not even doing like the math, right? It's not real math, let's be real.

Or it's like very simplified math. Just identifying those things, sometimes just that exercise, people are like, oh, I know what my decision is now. I think often just even thinking about this with that clarity, right, creates the resulting clarity then and think about all the factors. And there have been times where I've completely changed what I thought I would do. And like I can give an example. After I left academia, I was at a startup, and then following that, I started, like, my cute little baby startup company. I thought for sure I was going to go to a large consulting firm.

I mean, I had them ranked. I thought I knew my choices. And I was like, no, let's, let me think, let me identify what order I should go in. And it didn't matter how I rearranged all the numbers. Starting my own company was at the top. Let's go to that in your career. Yeah, so let's catch up. Exactly. So you decided you were going to do your PhD.

So I finished the PhD. I stay in academia for a while because I really like the research that I'm doing. It's really interesting. I think I'm kind of onto something and academia is a good place to do this, right? So I was a professor at Pepperdine for a few years. I go to Utah State for a few years. Again, like what's this opportunity? Pepperdine was a lovely place. The faculty were incredible. Malibu is gorgeous. Utah State, my alma mater comes along and they're like, we would like to hire you and we'll create an opportunity for a new position. So I took that pivot. And now it's a joint appointment to create an analytics program in the MIS department. I was

also an accounting professor because I have a master's in accounting. So it was kind of this like perfect situation. I was there for, you know, two, three years and I'm doing really, really well, a really strong path to tenure. Letter from the provost saying that things are looking really well. And I'm like, this is good, but it's not great. You had a spreadsheet that said that basically? This was in my gut. Oh, this is just a feeling. In my gut. I'm like, I'm doing

really well. My research is going really well. We just hit the Wall Street Journal because we find early signal that DevOps, like now it's being called DevOps, shows organizational impact. And so a few folks in industry were coming to me and they said, you know, this is super relevant.

You're really changing how we're doing things. This was still kind of early-ish in this research program. And they said, what if we create an opportunity for you where you could spend half of your time doing research and half of your time helping our company improve our engineering practices? And I was like, I think I might do this. You know, at that point, I just decided to go for it. And it was a little company here in Seattle called Chef Software. Oh, it was actually here in Seattle? I was here in Seattle.

Chef Software. Yeah, they did configuration management software. And also, that was a really interesting tie because, or kind of like pull, tug and pull, because I was doing this research with their competitor called Puppet, and I went to Chef. And so, I'm kind of like managing these relationships really well, or as well as I could. But that was also one of my first exposures to managing conflict at work and in professional relationships because this report, it was a research report, but it was done kind of through industry. So the main sponsor was Puppet, but I was working at Chef. And so how do I manage that?

So I was at Chef for a year and a half. And then at the end of that year and a half, we kind of looked at each other and we were like, I think we've reached the end of this road. Because I had done about as much as I could do for this little startup. They had about 200, 250 people. They really didn't need a researcher. They were doing me a solid. And then at the same time, we had kind of spun out a separate entity called DORA, DevOps Research and Assessment. And we said, so what should happen here? We had been doing research, the state of DevOps report.

But then so many companies were coming to us and they were saying, what if I want to have our own customized assessment? How could we do this? And, you know, I looked at a couple of my co-founders, you know, Jez Humble and Gene Kim, and I said, I know how to do this. I know the algorithm I would use. I know how I would build this out. I think we could do a SaaS company.

I have a very low risk tolerance. I have never wanted to start a company. And I think, you know, to your prior question, like, how have I thought about doing this before? I actually looked at them and I said, you know, who wants to be CEO? And they said, we think you should. And I said, I'll give you one year. And then I want you to tell me if you want me to continue doing this. And so we started this company. I mean, our first prototype, I drew pen and paper in the back

of a notebook and I showed it to Capital One. And we said, would you buy it if it looked like this? And then we just kind of iteratively built out pieces and pieces. And I was like, copy pasting pieces of it into reports as we went. And then at our offsite after one year, we read out results. I shared ARR and sales projections. And I said, OK, well, you have a first read refusal. And do you want to renew? And they looked at me like, what? And I'm like, no, it's been 12 months.

Do you want me to return again? And we just kind of decided if we wanted to. After that, things were going really well but we were also growing and scaling really, really rapidly. Too fast for us to keep up as a bootstrap company. And so, you know, I needed to figure out how to gain and acquire infrastructure. And so that would either have been external funding and hiring really rapidly or getting acquired. And so we, I approached a handful of companies and Google acquired us. Wow.

It's an amazing story. And must have also been a time of craziness and also fear of uncertainty. How, you know, what were some of your emotions during that time? It was, it was a lot. It was really interesting because it was kind of balancing also a few things, right? Because I had, I think some of it is almost balancing identity, right? Am I a researcher? And how do we think about maintaining that identity

and that credibility that I care so much about when the research and publication path, you know, kind of in finger quotes, that I care so much about has shifted, right? Because now I'm doing a lot of applied research and how do I think about that? Some of it was, am I an entrepreneur and what does that look like? And what does that credibility look like? How do I put out a product fast enough? How do I manage

and maintain this company? How do I manage and maintain relationships with my employees, with my partners, you know, with this partner ecosystem that we've developed. And then when we get acquired by Google, what does that look like? How do you manage that growth path? And these are all very, very different skills than you had as a software developer or even as a professor. Especially as a professor, right. And it was also a really, really wonderful time, I think, to grow and learn and iterate. And I'm really grateful for the partners that I had along the way, right? There were times that were bumpy, right?

But I appreciate that we were really honest with each other, right? There were times when we disagreed. And there were times when I also said, like, I know this is the right thing. Please, you have to let me do this. And there were times when I also said like I know this is the right thing please you have to let me do this and there were times when you know they had their expertise and then you know you were at the company the company gets sold to Google yeah Google now yet MSR and then I was at Google for a little bit and then I had this amazing opportunity to join GitHub. And I was really excited about that because that framing and that invitation, we were talking about, you know, what could it look like? You know, what would a perfect world be where I could return to something that looked like research and strategy?

Because I realized there were pieces about research that I really, really loved. And there are also pieces about, I don't want to say products, it's not quite product, but it's about strategy. What is it that I love doing in terms of kind of execution and identifying holes and how does that feed back into the research projects that I want to do? And so when Microsoft first approached me, they said, you know, what would your ideal job look like? And I kind of laid that out and they said, you know, well, let's think this through. And so when Microsoft first approached me, they said, you know, what would your ideal job look like? And I kind of laid that out. And they said, you know, well, let's think this through. And so we started with GitHub because, you know, if you want to study developers, that's an amazing place to start.

So we did a couple of iterations with the Octoverse report that were really rewarding. And then we said, you know, another good place, you know, another wonderful opportunity would be to think about developers and a developer experience lab. And if there's a research lab, where does that live? And we talked about it some more. Yeah. And we thought, you know, MSR really is the perfect place for that to live. And you've been at MSR now for a few years. And now we're going through yet another change. I mean, you've been through many of these changes. Before we talk about this current change, it seems like, again, coming back to this, you've been amazing of like when the environment shifts, finding out where to go, you know,

you have your spreadsheets, you know, as one mechanism. Now we're at another sea change with AI, right? And AI is clearly changing the way we write code, which is sort of in the innermost loop right now with GitHub, with our co-pilot. But it's probably going to make it much more into the inner and outer loop in the whole way we write code, in the way we develop with low code. So, I mean, first of all, how do you think about that sea change

and how do you deal with your research group and yourself as an identity again as the world around you is having this massive change towards AI? and how do you deal with your research group and yourself as an identity again, as the world around you is having this massive change towards AI? You know, sometimes I just laugh that the world is a circle, right? I'm really excited because we've come back around to getting to rethink what it means to do what we love, right? So I'm personally getting an opportunity to come back to be a developer again, right? What does it mean to dive back in and learn brand new things again? Because AI is new for almost all of us.

Even people who have been studying AI for 10 years are saying, like, so much of this is new. So much of this is something that we couldn't have predicted. I'm getting to, you know, dive back in and play with new tools and new technologies, and that I really love. I also love that you mentioned that, you know, there's the inner loop and there's the outer loop. And so in terms of my team, I'm really, really excited about the team that I get to work with because- Can you explain maybe a little bit just for the audience, inner and outer loop? Yeah, so inner and outer loop.

So inner loop is the coding that we do kind of locally. So it can be writing the actual code, local build. If you do local build, it can be debugging. It can be everything that's just right there on your screen as you're writing your code. Now, outer loop is everything that you do to get that code running in production. So it can be additional tests. It can be integration build, it can be everything out through release and deployment until we're operating that code and then continuing to operate that on our systems at scale.

I mean, the way I saw this so impressive when I joined Microsoft, of course, what it means to actually, that first of all, software development is a team sport. And also, it's not a team of like 11 people like soccer where I grew up with, but it's a team sport of, you know, potentially thousands of people. Right. And that there's actually there's no engineering systems around it. And then actually it's called software engineering for a good reason. Yeah, because there are systems around it that help us to scale

to that many people contributing to a single outcome. And so the inner loop is basically my notion is that I do my own little exercises with the ball and then the outer loop is when I actually you know do strategy with the whole team and see that I integrate well. Is that a reasonable analogy? Exactly and so sometimes the joke is you know what worked on my machine right that's kind of inner loop yeah and then outer loop is all of the orchestration, right? All of the architecture, everything else that we need to make things, especially really large things and complex distributed systems work at scale.

Yeah. And so if you think about how do you think AI would influence both the inner and outer loop? I mean, we see what's coming out in terms of, you know, GitHub Copilot and even more capabilities. I mean, in the Sparks paper, we described that GPT-4 can actually write an application with close to a thousand lines of code, right? So how do you think AI is actually going to influence developer

productivity and software engineering as a whole? I think there are so many exciting ways to think through this, right? I love your point that there's inner and there's outer loop, right? So, yes, we absolutely have opportunities to think about how it influences the way we write code, but I think it also has so many opportunities downstream, right? How can we use it to improve our code bases, right? Can it identify technical debt and clean up some of our technical debt for us? Can it help us think about downstream incident management? Can it help us manage our servers and our systems?

Can we use it to take a look at our code for us? Can we ask it, can you please help me improve the security posture of my code? Can you help me improve the performance of my code? Can you help me improve anything performance of my code? Can you help me improve anything else in my code, right? Even that explicitly, can you say, is there anything I haven't thought of in my code and what should I do? We can ask it, you know, to proactively watch and monitor our system's performance for us and then proactively manage that for us. You know, as we look, you know look even farther into the future,

there may be opportunities for brand new abstraction layers. What happens if we let LLMs or invite LLMs to execute for us and then reason about that code for us so that all we need to do is guide and direct it? We've been talking a lot about code, but maybe in the future, code will be sort be this lower level abstraction like what we have right now with assembly. There are very few people who still optimize, let's say, locks and database systems with the assembly code. But most people write at much higher level of abstraction.

So what do you see as this next level of abstractions that are coming? Is it language interaction? I do think some will be language. I really like that idea for at least a few things. For a couple of things I just mentioned, right? Like asking it to do things like, please check for security,

please improve the performance, can you help me generate workloads, can you help me run these type of canary tests, right? Like semantic linters with very bigger, much more bigger capabilities. Exactly. I also think there's an opportunity for graphical interfaces. And by that I mean, what if we create a diagram or

a UML and then ask it to implement that in the best way possible? Or reverse it, right? When I think about when I was working in codebases, the best way I could get a feel of the codebase was to code it, and then I could create this mental model. If we're doing less coding, how can we create that mental model?

I think there could be wonderful opportunities to ask or invite these LLMs to diagram some of these code bases for us. Don't just ask it to create documentation or explain it to us because language can be somewhat limited. But we know that diagrams and pictures can be incredibly powerful. We create the actual architecture diagram. Create an architecture diagram and create it in two or three different ways to help us understand how different components interact,

which can also help us understand where there may be redundancies in code. There's a huge amount of technical debt out there. So I think that kind of opens up some really interesting ideas for what could be there. There are also some wonderful horizons that we're already approaching in terms of testing and exploratory testing and what that can mean for really improving the way our code works. It's super exciting. I mean, I would love to be able to talk more about that with you. But let me ask one last question. I mean, you've had this absolutely stunning career.

I mean, if you think about where you started out, you know, then developer, you know, professor, startup founder, you know, working in a big company, being in a research, having a big company. For someone who's starting out, what's the career advice that you give for anybody who's right now starting and going into tech, people who are at university or just graduating? I think there would be two, and I think they're related. One would be assume that something can be figured out and that it's not hard. I think that would probably be one of my best tricks. I didn't find out until the end of college that girls were bad at math, which I am not, or that computers were

hard. It really helped that for most of my life, my dad just sort of helped me rethink through things or re-pivot or re-try lots of things. And so if it's hard, that's okay. It might mean that you should just spin it on its head and try to take another look. So that would be the first one. And I think the second one is consider new opportunities and go ahead and take them. And if after six or nine or 12 months, it's not the right opportunity, go ahead and change your mind. I would not be where I am today if I hadn't taken one or two really incredible opportunities that seemed a little bananas at the time, and it just worked out. That's amazing advice, especially also in something that tells us to A-B test even our own life.

Although the problem is that we have only a limited number of tests that we can do, but I mean, it clearly is an amazing story. Thank you so much for the conversation. Yeah, thank you so much for the conversation yeah thank you thank you to learn more about nicole's work or to see photos of nicole as a child in idaho visit aka.ms researcher stories