Software Misadventures - Bruno Connelly - Building and leading the global SRE org at LinkedIn - #14

Episode Date: September 12, 2021

Bruno Connelly is a VP of Engineering at LinkedIn. He leads the Site Engineering org responsible for LinkedIn's production infrastructure. He joins the show to talk about his journey in tech - from teaching himself how to code at a young age, building, maintaining and reverse engineering software as a teenager, building ISPs in the early part of his career (there are some fun stories that involve sleeping in the data center) to leading the SRE org at LinkedIn over the last decade. He talks about the early days at LinkedIn that involved a lot of firefighting to keep the site up, how the team built technical stability and scaled the platform. We also dive into how he grew the SRE org globally and overcame challenges that came with the growth. Throughout the conversation, he shares various nuggets of wisdom - like how to stay calm under pressure and how to make people feel at ease - as he describes his leadership style, people who have influenced him and what he thinks is a positive way to collaborate with people.   Website link: https://softwaremisadventures.com/bruno   Music Credits: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en

For me, you know, my prioritization is always, you know, the company and then my team and then myself. And I think, you know, when your priorities are in, you know, some version of that order and everyone else's priorities is in some version of that order, that's when things, you know, really start to work. And that's when you can, you know, much more easily put yourself in someone else's shoes and really kind of slow yourself down enough to think about like, okay, like where are they coming from and what's their perspective and what am I missing? Because if what I think is totally the right solution to this, they're not seeing, then I'm, I'm either not communicating this right, or they have a perspective that I don't have. Like I'm missing something obviously. So what is it? And I mean, so that for me has definitely been intentional to a certain extent of like getting more into that mode because it feels right. That feels like a great way to communicate and partner with folks.

Welcome to the Software Misadventures podcast, where we sit down with software and DevOps experts to hear their stories from the trenches about how software breaks in production. We are your hosts, Ronak, Austin, and Guang. We've seen firsthand how stressful it is when something breaks in production, but it's the best opportunity to learn about a system more deeply. When most of us started in this field, we didn't really know what to expect, and wish there were more resources on how veteran engineers overcame the daunting task of debugging complex systems.

In these conversations, we discuss the principles and practical tips to build resilient software, as well as advice to grow as technical leaders. Hey everyone, this is Ronak here. Our guest in this episode is Bruno Connelly. Bruno is a VP of Engineering at LinkedIn and leads the site engineering organization, which is responsible for LinkedIn's production infrastructure. He joins the show to talk about his journey in tech, from teaching himself how to code at a young age, building ISPs in the early part of his career,

which has some fun stories involving sleeping in the data center, and to now leading the SRE organization at LinkedIn. He talks about the early days at LinkedIn that involved a lot of firefighting to keep the site up, and some members of the team trained their spouses to take the alerts while they got some rest. He also shares how the team built technical stability and scaled the platform, which now supports 750 plus million members.

We also dive into how he grew the SRER globally and overcame challenges that came with the growth. Throughout this conversation, he shares various nuggets of wisdom, like how to stay calm under pressure and how to make people feel at ease, as he describes his leadership style, people who have influenced him, and what he thinks is a positive way to collaborate with people. We had a great time talking to Bruno and we learned a lot from him. Please enjoy this fun conversation with Bruno Connelly.

Bruno, it's such a pleasure to have you with us today. Welcome to the show. It's a pleasure to be here. Thanks for having me. So while we were researching for this episode, we learned that you started working at a very early age and you came from a non-traditional background. So we thought we would start at the very beginning. Can you tell us how you got started in tech? I'm happy to. I'm also now curious how much research you guys did and how many surprises I have in store for me. Not a lot, just some. So in terms of, I'll go way back if you guys don't mind, in terms of like kind of my intro to the industry and infrastructure and frankly kind of being an SRE. Like, I like to joke that I've been an SRE since I was 10 years old.

And while it's a joke, there's kind of some truth to it. And it would probably be more like maybe 12 or 13 years old. And so the kind of condensed version of this is, you know, like most kids that grew up at the time and in the region that I did had kind of passing exposure to computers, like the Apple II, the Atari 800, the C64, the TRS-80, you know, just kind of here and there, everything from like in school classrooms to like in a mall, in a like electronics boutique or, you know, one of those.

And I think when I was about, and I was always that kid that was like drawn to the like, I don't know what this is, but it's super interesting. And I need to know more about this. And when I was about 10, my uncle was a computer hobbyist at the time. And he had this room in his house that had a C64, also an Atari ST, a bunch of MIDI keyboards that he had connected to. If you remember the Atari ST, one of its selling points was the mini interface and a modem. And then like big stacks of software. So that was like the first opportunity I got to really kind of get pulled in to firsthand exposure,

as well as just like the opportunity to experiment and play with these super fascinating devices. And a couple of things one uh sierra games was uh he had like this big box of uh you know questionably downloaded software um and one of those things that i i stumbled upon like just trying these things out was police quest and police quest like totally just like immediately sucked me in i don't know if you're if either of you were familiar with the Sierra series. They took the approach that text-based adventure games had done and then layered on a graphical aspect where you navigate a character through a world and then type

what you would have the character do to go through the storyline and solve puzzles. They were just super well done, super immersive. Ken Williams, who founded the company actually just, uh, just wrote a memoir of sorts. I'm trying to remember what it was called. I think it was not all happy, not all fairy tales have a happy ending, um, which I would, it's a recommended read. Um, I partially for me, like it's, you know, given it just has so much tie into my intro to computers, there's a lot of nostalgia for me, but it's, it's also a great read. So anyway, so those games really kind of pulled me in and then the kind of next step from that

was the modem and the bbs scene at the time and uh you know initially bbss were just like an entry point for me to find more software um but very quickly turned into like this whole other world of you know effectively like super early online communities, people contributing to message boards together, electronic mail with each other, uploading and downloading software. So, you know, long story short, I got very involved in the BBS scene and, you know, ultimately ran my own BBS, which then, you know, got into, you know, taking software that someone else wrote, you know, kind of reverse engineering it, extending it on my own. That's effectively where I, unbeknownst to myself at the time, taught myself to code.

Slight snippet, there was, I don't know if I'm remembering these details correctly because it's just been so long. On the Atari ST, there was a BBS software package called, I think it was BBS Express ST that had basically like a scripting, like slash macro language built into it that you could extend and like build online games and doors and whatnot. And, you know, I remember thinking back like 10 or 20 years later, like, oh, that was my, that was like my introduction to writing software.

And I didn't even realize it at the time. So long story short, you know, I spent, you know, probably like from age 12 to 16 to running, you know, a BBS that, you know, got progressively bigger and larger and had, you know, a larger audience. And as I could save up money in any possible way, I would, you know, upgrade my 1200 baud modem that I got from a neighbor that had like sitting in a closet to like a U S robotics, uh, career HST that, you know, set me back like 500 bucks at the time. Um, uh, getting a hard drive, et cetera, et cetera. Anyway, you know, I ran this online service, um, that like had to be up and available for folks. And I'd like got, I took it very seriously

and got very immersed into it. So from there, that led me to ultimately starting an ISP with a small number of folks, building a regional ISP, later working at a much larger regional ISP in San Francisco, which was a blast. This company called SlipNet. I think we brought some of the first DSL to the Bay Area at the time, definitely to the city of San Francisco, and then ultimately into consumer internet and where I am today. So it's kind of funny. My path has been almost kind of linear. And I like to think, and of course I'm biased,

that the ISPs in the 90s, in the early 90s, and the early days of dial-up internet, was some of the formational and proving grounds is the right word. But that was SRE work. You were an engineer that you were a software engineer, you were a systems engineer, you were a network engineer. You were punching down pairs of copper on punchdown blocks, like you had to literally kind of do a little bit of everything

to build and maintain a set of infrastructure that other people rely on that you want to make as resilient and available and performant as possible. We heard some rumors about you were sleeping in the data centers. Which phase of this did this

happen? Yeah, you'll have to tell me your your uh research sources we are a very professional you know podcasting we do not disclose you know those sensitive information right so yeah there was this picture that like someone was following me and i don't really even know how um you know thanks to social networks and whatnot, these things now live on. So I worked at a company called Digital Island in the late 90s that was also like global ISP, like later turned into an early CDN play. And part of that is like we built a bunch of data centers all over the world. And I was super fortunate to be able to be a part of a bunch of those.

And there was one, I think it was in London in the Docklands, I don't remember, where we were just, you know, working through everything we needed to do and, you know, kind of working around the clock because we were young and silly and enjoyed it. And when it needed to take a break i built myself a bed of out of cardboard boxes from like sun you know e220s or whatever it was that we were building that data center out with and somebody snapped a picture of me crawl up sleeping on the stack of sun cardboard boxes that i fashioned into a bed and that picture is apparently still on the internet.

We're going to need to look deeper because when we were researching, we couldn't find this picture of your sleeping on a cardboard box. But it will be featured picture now. So yeah. Yeah, it's probably not indexed, but I'm happy to share a copy with you. Oh, yeah, that would be great. So today, you lead the site engineering organization at LinkedIn. And you joined when the SRE team was seven people, which was more than a decade ago. Can you talk more about how you came to join LinkedIn and what the early days were like? Sure. Yeah, kind of a few things to unpack in there. So in terms of coming to LinkedIn, I was previously at Yahoo. I joined there via the to me acquisition and was at Yahoo for seven years. And like, it was just an amazing, fantastic, informative experience. And that's where I got

so much exposure to everything from, you know, thinking about scale in ways that I hadn't previously. Amazing infrastructure engineering talent and just engineering talent in general, as well as just like really grew my network and met a lot of folks that really, you know, continue to play important roles in my career. And at Yahoo, there were two projects that I, you know, felt super, I still feel super proud to have been a part of. One was, I think we called it SearchX. I actually don't remember. It's been so long ago, which was taking the Intimate search engine backend, bringing it into the Yahoo environment, and then migrating Yahoo's search traffic, which I believe at the time used Google, you know, as an OEM provider on the back end. We moved it to the on-prem search engine back end. And the second was what was called Project Panama internally,

which was like a complete reboot of Yahoo's search advertising platform. Everything from like the advertiser tools to advertiser features, I think the introduction of ECPC, stuff like that, to new ranking and relevance mechanisms to a complete rewrite of the backend storage systems where previously it was a monolithic Oracle instance that was globally replicated to an internally built key value store that was optimized for the ads,

certain days, data set, et cetera, et cetera. So within that second project, I met this gentleman named David Hankey, who has been on your show. And your laugh is appropriate hearing the utterance of his name because he's just such a force of nature and like such a like not only a great human being but just like an amazing like a huge personality that like you know he's very charismatic and immediately kind of pulls everyone in anyway um so david led that project for yahoo massive project and that's where um i got the chance to get to know mr hanky uh and um you know long story short, he retired from Yahoo, and I assumed he was retired and done. And I remember seeing, I think, on LinkedIn,

that he showed up at LinkedIn, which was super interesting to me, like, that he was working again. So I think I, you know, reached out to him at the time just of, you know, hey, like, super surprised that you're working, like, what is it that pulled you in? And like, let's talk about it. And, and as I spent more time with him, uh, you know, that's where I, I got to understand everything that pulled him and everything from just like, you know, super excited about the, you know, not just like the LinkedIn product itself, but the possibilities of the LinkedIn product. And like where he and, you know, and Jeff Wiener at the time and others read, of course, you know, we're going to take the platform as well as just the technical challenges that existed, the scale that was in front of them. And I mean, to be super

honest, some of the like technical concerns and like stability concerns that they had to get to that scale, which was a big part of why David was there. And so, you know, while I was, I loved Yahoo and was super, like I said, I had such a great experience there. It seemed like this kind of unique opportunity to be a part of this, this platform that, you know, the mission and vision absolutely like resonated with me and seemed, seemed amazing coupled with the technical challenges and scale that would be in front of it and the opportunity to do that, you know, a little more, you know, on my own and like go through some of those learnings on myself versus, you know, Yahoo was a, you know,

I think probably a 5,000 person company by the time, you know, the ink to me acquisition joined and et cetera, et cetera. So that's, that was my intro. That was how I landed at LinkedIn. The other part of that question was like early days at LinkedIn? Is that? Yeah, we would love to dig into the early days at LinkedIn. However, one thing which I would like to know is when you were coming from Yahoo to LinkedIn, Yahoo was a much bigger company at the time. And when you joined LinkedIn, were there any surprises, like things you weren't expecting

or things you expected to be different? I imagine the short answer is probably yes, but also kind of no. I'm also guessing that your research probably has some surprises you want to talk about, perhaps. I mean, so here's the thing. So on one hand, like, yes, I think the, and I, you know, don't hold me to these numbers. I'm doing my best to kind of remember. I think the, you know, the whole company was less than a thousand people. I think engineering was like maybe three or 400 people. The, all the entirety of LinkedIn was served out of a single data center on, you know, probably sub a thousand computers. You know, it was definitely different than what I had seen at Yahoo. I guess, you know, one other thing I just want to mention about LinkedIn that was kind of interesting was, you know, LinkedIn had a really long kind of growth curve and, you know, before it hit its knee and its growth curve. And it wasn't like, you know, LinkedIn existed and then, you know, it started scaling. We had to figure this out.

The, you know, there was like an MVP period of the product that was probably like 2003 to 2010, where things really started to go, um, uh, uh, you know, kind of like reached an inflection point and went nonlinear in terms of growth patterns and like member signups and traffic and all this sort of stuff. So I think that was kind of unique to LinkedIn in the sense that there was like a long period of like iteration and, you know, that's to a certain extent, technical debt that just accrues naturally while you're trying to build a product and, and try to find market fit for it. So, you know, no fault to anyone other than that's just kind of how these processes work. So with all that, you know, I would say like, I don't think I was,

you know, I also, you know, work in the industry long enough at that point that I know how the sausage is made. And I know that there is no such thing as perfect infrastructure or perfect environment. So I think if anything, the surprise for me would just be how fast things were growing. And we were seeing record traffic numbers day after day after day after day. So that was probably more of a surprise to me than anything else. The rest was just like, that's what comes along with scaling stuff on the internet and building complex software platforms with hundreds of people working in concert to build them. Like that stuff is, it's not easy and it's not perfect. That makes sense. Now, being an SRE at

LinkedIn, I've heard the stories about what the early days were like for the side ops team, especially when LinkedIn was going through this growth phase. Can you describe what some of the challenges were at the time, and especially what the life was like on the side ops team, and the kind of things you would see on a day-to-day basis given the technical instability sure um uh yeah there's there's a lot in there um i would say the so back to the point i mentioned that you know one like things were just growing really quickly and i remember we were literally uncovering new capacity bottlenecks um uh you know multiple times per week, everything from like, you know, switch up links to like, you know, like, like literally like, oh, like, yes, like we're now, you know,

maxing these things out in ways that we didn't expect to previously. So that was, that was part of it. The other is that, you know, there was an element of, you know, LinkedIn had taken a pretty traditional operational approach historically. So, you know, way, way back in those days, it was, you know, ing was ing and ops was ops and certain people only had access to certain things. And, you know, some of that in the worst, in the worst cases would result in, you know, the ops teams, which was, which was my team at the time, carrying a lot of, a lot of weight and dealing with, you know, the ops teams, which was my team at the time, carrying a lot of weight and dealing with, you know, all of the outages and all of the operational work. And, you know, frankly, sometimes being put in situations where, you know, they're probably not even the right,

you know, first team to be involved, you know, but just kind of, you know, servicing more as like a 911 dispatcher than an engineer that can really kind of help. And so, you know, because of that, and, you know, this, you know, you've probably heard some of these stories, and I've spoken about some of them publicly in the past, the most famous one, I think, is that at the time, there was a pager. And I forget what the device was. From the stories I've heard, I think it was a BlackBerry. It might have been a BlackBerry. Yeah, that folks literally, like, if you were on call, you would get the BlackBerry. It might have been a BlackBerry, yeah. That folks literally, like if you were on call, you would get the BlackBerry. So you would like physically hand it off to the next person.

And this thing like literally didn't stop going off. And I'm not like hyperbolizing like, oh, it was, you know, it was like literally like once a minute, there would be a new alert. And, you know, there would be some level of like triage, you know, and folks would do best effort just given, you know, how often the stuff was coming in. And some folks said like the story, you know, some folks like literally had helped, you know, train their spouses and partners to help them be able to triage alerts since they came in so they could, you know, get a break. And, you know, their partner would let them know like, hey, this actually sounds pretty bad.

You know, there's a problem. There's alerts from, you alerts from graph latency again. Like, it looks like this might be gnarly. You may want to wake up and take a look. So it was, there was a lot going on. Wow. Well, that sounds really rough. And I think the team was seven people at this time.

When you joined, were you managing this team or were you part of the team and on the on-call rotation? So there were, yeah, so I inherited a seven-person team. I don't think I was ever literally in an on-call rotation. But I mean, we were all on-call. Like there was another thing that happened back then. was called i think it was called the 6 a.m shift where um if you were like you know back in those days like the traffic peaks would start around um around 6 a.m and keep in mind like the platform was seeing record traffic levels every day and like finding new you know um capacity issues and bottlenecks every day so part of i think if it was i can't remember if it was, I can't remember if it was

separate from on-call or if it was part of being on-call either way, there was a 6am shift and you would come into the office at 6am and be at a keyboard. So you can be ready for, you know, as the platform, you know, starts to fall over in the, in the backend and help, you know, kind of put Humpty Dumpty back together again a little bit. Um, so it like the first off like i want to give credit to that team like and many of those folks are actually still at linkedin today like fantastic fantastic people on that team and like you know i don't want to name too many names i don't want to leave folks out but like tony kwan and bosky devaraj can't come to mind like both of you know we're still engineers at linkedin today um. It was a fantastic team.

It was just like there was just so much happening. It was more than that team could really kind of handle on their own. So yeah, definitely I spent a lot of time in the trenches, but I don't think I was ever technically in the on-call rotation. Gotcha. And things changed, obviously, considering this was such a crucial time for the company. What were some of the key decisions that were made that changed LinkedIn's course and brought in that level of technical stability to take it forward?

Yeah, so I don't, I actually haven't listened to your episode with David Hanke. So I don't know which of the stories he talked about. Actually, well, I'm super curious to hear that with David Hanke. So I don't know which of those stories he talked about. I actually will. I'm super curious to hear that conversation he had with you all. A few things come to mind. One is David and, you know, to a certain extent, Kevin Scott, who joined LinkedIn's head of engineering shortly after David and ultimately took over for David when he retired.

Both of those gentlemen played a huge role in the side of transition at LinkedIn. And most of it absolutely goes to David Inkey, like full credit to David. You know, part of it was like this this intentional transition to a side of culture, a belief that, you know, everything always starts with side-up, every decision starts with side-up. And if there's ever a prioritization conversation or concern, and side-up is one of the options, then the answer is side-up, which was exactly what the company needed at the time. And sometimes that meant pivoting decisions and frameworks from, you know, is the, again, keep in mind, like the company went through this long MVP period. So there was definitely a lot of focus on like product fit and like,

actually product is most important because, you know, if the product isn't right, then everything else can't be built on top of that. But, you know, this was now a different chapter in the company where actually, you know, signup comes different chapter in the company where um actually you know side up is uh comes before some of these product decisions and if that means we need to delay the you know the ramping or whatever it is of this this product feature that we know is going to be amazing we're super excited about like that's totally the right call so that um that was, I would say, one of the biggest transitions that comes to mind. Yeah, David talked about site up, and it is something that I can attest to being an SRE at LinkedIn. And if I remember correctly, LinkedIn was trying to file for an IPO around that time.

And there were some questions whether to file for an IPO or not, given the issues with the site. Can you speak more to that? And also, like, what were some of the core principles that laid the foundation for SRE at LinkedIn? Yeah. Sorry, you're triggering some of those early days in scaling and the IPO, you know, one thing that I'll just share this briefly that comes to mind is I remember Hank, he was actually, he were in the kind of final decisions of, of the S1 filing. And we actually, um, uh, there was like an Oracle backend at the time that we were having, that was reaching scaling limits. And I remember, um, you know, we actually had to, we had conversations about, you know, like, does the S1 filing make sense based on, you know, where we are with kind of understanding

this, this level of capacity issue. Very different world now. And to kind of segue that into your other question about the intention and direction of the SRE team and how that changed over time, we actually had three founding principles that we leveraged when we started the team that carried us quite a ways. And the first was that side of bizarre is our top priority. And I mentioned this earlier, this was really that kind of converting the culture of the company,

you know, and the mindset of the company to, to a side of culture. The second was what we called empower developer ownership. And that was, you know, both on one hand, you know, getting out of the mode of like, Hey, we want to like, you know, both on one hand, you know, getting out of the mode of like, hey, we want to like,

you know, the ops team has to be the team that owns and operates all this stuff. Like, no, we want to empower any developer at the company to be able to do that. And keep in mind, you know, a long time ago in the industry as well. And there was even, you know, some of this mindset of like, oh, well, you know, can do developers know what to do in production? Are they going to, you know, it's like, it's like, well, hey, like one, I'm not actually concerned about that. And two, even if I were like, let's do it in a way where we can empower them to do it in a safe way. But then, you know, that also, you know, then just pivoted into, you know, a culture of really, you know, wanting any engineer at the company to take great pride of, you know, ownership of what they're building and what

they're scaling and production and how better to do that than to like give them as much control as possible. And then the third was the operations as an engineering problem and treating operations as an engineering problem. And that, you know, meant everything from, you know, looking at how we're hiring, making sure that we're, you know, hiring folks that are generalist engineers. Obviously, a focus on automation, you know, which is at the, you know, near and dear to all of our hearts. But, you know, it also meant, you know, pivoting how we think about designing software towards things like resilience. And, you know, that not only operations and, you know, you know, toil related work as it's often referred to these days can and should be automated, but also that like, you know, but also just kind of carried us a long way and played a large role in everything from defining the culture to like the types of folks that we

hired, to even just kind of, you know, how the team has evolved and continues to evolve over the years. So these principles make a lot of sense. And I believe you were trying to build out the SRE team back in 2011, 2012. Now today, the SRE function is relatively well understood. However, I don't know back in the days how well known it was. So were there any challenges in hiring people to join the SRE team, given how known this function was at the time? Yes, there were definitely all those challenges were real. 2011 like it wasn't even like you know outside of google who obviously um uh you know coined the term uh literally um and you know defined much about it um that's i remember like it was it was

used very inconsistently at the time i remember zynga had an sre team that i believe like that was literally their knock and they called it sre um a very like tier one focus team um facebook had was using the title at the time that was before they pivoted to production engineering. Um, so like, yeah, it was like super inconsistent across the industry as well. Uh, so like that was one problem. The other was, I think some of what you were alluding to is just like, it wasn't well known, um, or, uh, you know, it wasn't well understood.

And there was like this perception that folks had of operations that we were also countering. So like all of those were real problems at the time. Also, you know, LinkedIn at the time didn't have the hiring brand either. So we had to get creative with a lot of just, you know, how we attracted and hired folks.

Do you recall what were some of these creative ways that involved hiring people? I want to think about that a little. I think the only thing that comes to mind was we just got really creative about how we targeted folks and just like trying to find environments where it's likely that there are SREs that maybe don't even know they're SREs. And that's everything from like folks that were in ISP environments, like I mentioned, to, you know, folks that were running large networks at their university, to folks that were like full stack engineers and smaller shops, but really gravitated towards some of these problems.

That's where, you know, our sourcing and recruiting team at the time got super creative of, you know, like finding more avenues where these folks existed, where, like I said, they may not even realize that that's what they are. That's pretty neat. Talking about building teams,

you've built out teams globally and twice now once at yahoo and then at linkedin we don't speak with a lot of people who have done this before so we would love it if you could share your experience uh building our teams globally and like what were the challenges that you faced and if you have any advice for teams or companies who are trying to build out engineering presence in a new country like the kind of things that they should think about and they could do to set up the team for success yeah so there's there's a lot in there um i so i've had the opportunity to do that twice to point it both at yahoo and at linkedin and um in both cases built uh teams in bangalore

actually in india from the ground up and the the yahoo team specifically um i i had the opportunity to build an inner a team both in bangalore and actually a team in beijing and that was like a whole new world of learning especially especially back in those days, this was like probably 2005 or 2006. Uh, uh, I remember I was at, for the India team, I was making job postings on my own, um, to like, I really liked the, like the Pearl mailing list and just like all sorts of like, you know, you know, weird places where I could, you know, try to uncover folks that, um to uncover folks that would be interested in fits for this type of work. And, you know, that was, especially at that point in time in India,

infrastructure engineering wasn't near as like that. The Bangalore engineering scene had not kind of, you know, moved into that space yet. So, you know, the hiring, I remember staying up until, you know, 10 or 11 PM to phone screen folks to, to just try and build that initial team. So, you know, that, that was a set of challenges on its own. The, you know, both in that case, as well as in the case of LinkedIn, when we did decide to grow internationally into India, there were a few things that, you know, one was just being able to tap into another hiring market and have the opportunity to take advantage of a hiring market outside of the Bay Area at the time. And then, of course, you know, the benefits of geographic distribution, everything that comes along with that. And, you know, with that also comes the challenges of how do you keep that team super connected to your point and really kind of identifying with the culture and feeling like a part of the team

and feeling like they're just like connected to decisions. And, you know, it's easy to think about now, but like, you know, put yourself in the position of those folks where if they have a question, you know, that, you know, one of us would just, you know, poke a head over a cube or, you know, nowadays ping someone on Slack, you know, they've got to wait 12 hours to, or longer to get a response because they've got to fire an email. So it's like, how do you, how do you account for that and how you communicate? And even like, you know, as you're writing emails to each other, like try to predict the next few questions that your colleagues in another office may ask. So you can answer it ahead of time and, you know, really kind of shifting the culture to that. The other is making sure that those teams have an identity

and that they have something that they own and have some level, probably a level of autonomy of what they're committing, their contributions to the company, and obviously not the follow the sun support team type of thing, which I don't think know think anyone is like it's you know that ship is sailed but at one point in time like there were you know that's how you know parts of the industry would look at at remote teams or international teams were you able to find someone that's based there that kind of help you with you know some of the more logistic stuff that

you can really trust was that like a pretty like important person that you were looking for? Or did you like do a lot of the stuff like by yourself? So in the case of Yahoo, there was already there was an engineering presence, a small engineering office there. So there was a gentleman there that that helped make that happen for us. The case of LinkedIn, our first chief security officer, GK, who also was chief parent at Yahoo for a long time, he built that. He was that logistics person for us. So he built our Bangalore office from the ground up. In both of those cases, I think having someone there who really understands all aspects of the company, including the culture of the company is a very important part of that being successful. And GK absolutely played that role. And then otherwise, you know, for me, it was like a big part of it absolutely was identifying, okay, who are the leaders that

we can build around to grow the rest of these infrastructure teams for us. But definitely that, you know, that seed person that understands the company, I would say is definitely a key element if you're lucky enough to have it. Thanks for sharing that. That's super valuable. Switching gears a little bit. So considering your career in site reliability engineering, I would imagine you've seen a lot of incidents, a lot of outages in your time. And some of these situations are extremely high stakes. They are crucial to the business and there's a lot of pressure. And in what I have observed and in what many people have mentioned to me at LinkedIn is that when there is a situation like this, you are, if not the calmest,

one of the most calm people in the room what i would love to know is how have you developed this ability and what are some of the practices that work for you in staying calm under these high pressure situations uh it's a great question i love it um i think you know some of it is you know super honest it's just like some of it's just my know, super honest. It's just like, some of it is just my nature. I think everything from like genetics to like, you know, adaptations from my childhood or, you know, who knows what, that it is, some of that is my nature and my approach to life.

And, you know, kind of what feels right to me. Part of it is I've been doing this a long time. And, you know, as you know, you've also been doing this a long time. Like you, the more you see the way complex systems can and do fail, the more you recognize the patterns and the more you reach, you know, a certain level of comfort and confidence of knowing, you know, how you navigate through these things, as well as knowing that like some of it is, it's part of the process, right? It's like an exhaustive building some of these systems, especially when you're moving as

fast as we are. And with, you know, so many folks, as I mentioned earlier, like building a concert to build these complex things, like it is, it's an exhaust of the systems. And, you know, the thing that always, it always kind of grounded me to is like all these systems, we built them. And, you know, the like literally 100% of the people that made the decisions and put these things together, I have access to direct access to. So what better team to figure out why they're having an issue right now and figure out the right way to approach it. Coupled with, and at the end of this, I know the learnings we're going to get, like no matter how stressful and painful it is in the moment, the learnings in the end are going to be amazing and are going to make us a better team, a better

platform. And, you know, so like to a certain extent, like I embrace the outages, not because I love incidents or definitely don't love outages, but the, you know, like I said, all that there from their part of the process to they will make us better, there will be fantastic learnings from them. And I know that I don't believe in the apocalypse until the apocalypse happens. And thus I know we will navigate an end to any incident we find ourselves in. I definitely don't have enough street cred on this but from the few times where i did sort of have to be in kind of a live uh situation where there's incident happening and then trying to triage i feel like the one part of it is definitely the technical like hey you know

like trying to figure out like this is a complex system how do we do it but then the other part is also like people right like at the end of the day you're with different people. It's the people that wrote the code. Different people understand different parts of the architecture better. I am curious to get your thoughts on... There are definitely times where I'm like, oh man, this is such a tough problem, but the people is great. And we're super... Really enjoy working with each other.

And then they were like, I'll end this together. But then sometimes it doesn't quite happen. So do you have any tips or advice for working with each other and then they were like all in this together but then sometimes you know it doesn't quite happen so do you have any um tips or like kind of advice for like kind of working with people that maybe take a very different approach or people who are like maybe just very like now volatile maybe that's not the right word but you know like how do you kind of deal with different uh personalities during those situations? Interesting question. I guess my experience is that folks are feeling volatile, to use your term. It's probably because they don't, you know, feel, you know, confident or comfortable in the, you know, the overall control of the situation. So I think, you know, one of the

things I've always tried to do is to just kind of help be, you know, a bit of a grounding force for the team in terms of, you know, my role historically in those things is really just, you know, try to ask good questions and, you know, and encourage others to do the same so we can get all the data in one place and get closer to a point of control, as well as just separating the team up as needed, getting people focused into different areas and picking three or four different areas and putting teams in those different places so you can break bigger problems into smaller problems, right?

And that, you know, I think if folks are, you know, struggling in some of those situations, it's probably just because they're overwhelmed, right? And it's like, it's understandable because it's an overwhelming thing. So if anything, my role is to like help break it into smaller bits so it can be less overwhelming including less overwhelming for me as well because it's the whole thing can be overwhelming yep and i can see that being kind of more you know being rationale i think it would work for like maybe more peers or like people that are reporting into you it's like hey you know let me kind of take a step back. This is what's going on. And, but like what happens if it's like you're,

you know, someone that's, so I'm like slowly kind of working towards the question. So when we're chatting with Enki, he kind of mentioned that there was this kind of priority zero event, things that not, you know, going super well, where you actually, cause at that point you were reporting into him, but then you

asked you know him to actually leave the room um so i think he was very comfortable sharing sort of the the context but yeah i would love to kind of just hear more about what happened how you deal uh dealt with it and things like that yeah it's it's funny because i do i i remember the incident but it never really it never really registered with me like it's not you know how you how you have some memories that really kind of sear themselves into your brain? It's not one of those memories for me. It's just one that I happen to remember. Just another day at the office, you know. But I've heard Hankey tell this story multiple times.

So apparently it is one of those memories for him where it did kind of sear in for him. I think it goes back to like i said how i see my role in those which is to like help the team feel calm and in control and like we're breaking things into into smaller problems and um and in the case of hanky he's uh you know he he there's there's one core difference i've shared this with him in the past too, between he and I and the, I would, um, the things where I would feel,

you know, the most uncomfortable and, uh, uh, you know, maybe I guess like have anxiety about, for lack of better terms, historically would be like people related issues and like,

um, uh, you know, uh, like man, you know, managing senior folks and larger teams and dealing with disagreements. Like that's where I would get more anxiety. And he was a huge source and mentor for me of how to, how to grow and manage in those situations, the technical bits and the incident

bits back to my previous point, like those were never really high anxiety things for me. I actually felt pretty natural and comfortable in managing those. Hanky was the opposite. He was the, you know, super comfortable with the people parts and like, Hey, like we can, you know, any, any, you know, whatever the problem is, we can, we can solve it rationally, but would get much more anxious about the technical parts. And so in this specific situation, you know, Hanky definitely had, and I don't remember the exact outage, but I think it was like an actual, an outage, like, you know, the platform was down. And he, you know, he was concerned as he should be and in the room. And I, you know, I saw it was kind of reached a point where, you know, while his intentions were, you know, you know, nothing but, but perfect,

the result was, you know, just bringing like kind of more anxiety and a little bit of chaos into the room. So I, you know, you calmly pulled him to the side and said, and, and I, I think I asked him to leave the room and he, and he did. And you know, I explained like my approach to the situation that I, you know, like this is this is the path that we're going to keep taking it. Are you comfortable with that? And this is my commitment to you of how I'll keep you in the loop and up to date. Does that all sound good? You know, would you like to change any of that?

And with that, you know, my ask is that you, you know, you stay outside of the room. Just give me the space to run this so I can keep the chaos to a minimum. And he was super cool with it, of course. But yeah, I mean, sometimes I get Hank and I had a lot of history, obviously still do, so I knew him very well. So it's probably easier for me to have that conversation. But I definitely get that that can be a tough situation for folks but i think it's also like a very reasonable thing to do and if you feel that it's the right thing to do

um you should absolutely do it in a in a clear and respectful way the the first thing comes to mind if i'm about to do something like that is like the table flip uh uh gif that's sort of replaying in my head if i you know ask uh ask that but i guess to your point is that do you think it's like the trust like you know when like that you the both of you have built over time such that you know that when you approach him you know being polite but then very matter of fact hey this is what's what i'm observing that he trusts you enough to kind of believe you know your judgment of the situation and then kind of um take your advice yeah yeah um yeah i mean i think it was a mix of absolutely you know he had you know trust in me you know trust in terms of like you know consistency over time he knows me and how I operate. And also, you know, it's like, you know, you say the table flip thing.

I feel like that's not, you know, obviously no one likes incidents. No one likes outages. You know, like at some point, you know, way back in our history as infrastructure engineers, you know, you would hear stories like, you know, people getting fired over incidents and things like that. Like, that's not how, that's not how the world works, at least not anymore, in my opinion. And so I think that, you know, that's at least, yeah, I think there's an element of that as well. You know, if we're all coming from a place of, we're aligned on like getting the company's platform back to a place where it needs to be. Like that's the core alignment point, right?

Everything else building from that is building into a good space. There's a lot to learn there. Thanks for being open about it. Talking about leadership, what I would love to know is who are some of the people that have had an impact on you who have influenced your leadership style and people that you've learned the most from? Sure. You know, probably like three individuals come to mind for me. And there's so many, so many more. I've been so lucky to work with so many amazing folks and learn from so many amazing folks.

So three that just happened to come to mind. One would be Cheryl I know who I had the chance to work indirectly for her at Yahoo for a good period. I think she might be at Walmart now. And she had this, she took over the infrastructure teams at Yahoo at one point, and it just brought this level of engineering rigor. And like she was, her background was, I think everything from like in her early days, medical devices, and then, you know, the application side of consumer internet and um she brought a lot of you know engineering rigor and to a certain extent just like um you know accountability through um you know frameworks that we could

that really kind of up leveled our our abilities in our game and definitely i took a lot of that to heart and she was a fantastic mentor for me. David was absolutely one. And I think we've talked about him quite a bit, but he obviously was a big influence on me. And I mean, I can't even begin to think of all the things that he taught me about leadership and how to partner and influence and collaborate with people.

And then the third would be Kevin Scott for certain. Kevin, I mentioned, I think, a little bit before, was he was, I think, pretty early at Google, worked on, I think, ads quality and AdWords at Google, and then went on to AdMob and then back at Google and then ended up at LinkedIn at that point. And, you know, Kevin, when he joined, like, from his experience at Google alone,

not to mention building and scaling AdMob, he just understood. He came in with this worldview and understanding infrastructure at scale in ways that, you know, few of us did. I definitely did not. And on top of just, like, being, like, a pretty impressive human being. And he helped. I don't quite know how to articulate it. Um, the, I guess he helped me kind of understand that, you know, sometimes like as a manager, you almost feel like, you know, that, um, you know, these are the rules that you have to follow. These are the things that you have to do. And like, maybe it's, you know, not the best

decision for the team, but it's like what you need to do as a manager. I don't know if that kind of resonates. And he taught me that that's like, that's not true. Like, like, you know, doing the right thing for the company and doing the right thing for your team and et cetera, et cetera, like that you can, you can have all of those things. And so I don't know if I articulated that very well, but that was pretty eye-opening for me as a manager. Like it's, you know, sometimes like, you know, the rules as you interpret them, if it feels like it's probably the wrong thing to do

for some other reason, then it probably is. And we should do it differently. And so those are the three individuals that come to mind for me. Nice. So zooming out a little to get your perspective on the growth of the SRE team at LinkedIn, what were some of the growing challenges? I mean, after a point, it's not possible for you to know every individual.

So what were the challenges that came with it? And as you reflect on this journey, what would you do differently? Or another way to think about this question is if you were to call yourself from 10 years ago, what advice would you give yourself? The, so in terms of growing the team, you know, you mentioned I started with seven folks and, you know, that that was sometime in 2010. I think by the time maybe 2012 or 2013, we were well into the space of growing the team very quickly and hiring a lot. And somewhere in those years, I forget which ones, I literally interviewed every single person that joined the SRE team or like interviewed for the SRE team. Like I was

literally on every single panel and, you know, was doing probably a hundred interviews a year. And because of that, you know, and this was when we really started ramping up and growing the team. Because of that, I literally do every person on the team and, you know, had the opportunity to make like an initial connection and get to know each one of those folks. And my, my leadership style at the time, and I, you know, it still is, is maybe parts of like what people sometimes call management by walking around, you know, definitely like leveraging and not like, you know, checking on like quality or performance or anything like that, but like, like building and maintaining connections with the folks and with the team. And by, by, you know, through that, by, by virtue of doing that, also getting just like amazing insights about, you know, how, how the team functions and what's working and what's not and where, like, you know,

I get so many data points about how to, to, you know, grow and, and, you know, continue to chart the, you know, like make decisions about the team. And I definitely, you know, reached a point where, you know, it was probably like Dunbar's number or something like that of like, you know, reached a point where I couldn't scale that anymore. And it was, I think Dunbar's number is like 150, somewhere around there, whatever it is, the, the, the, the, I'm not going to give the, be able to do justice, but you know, the, the, the, the belief that the, you know, the human brain can only maintain so many human connections and above a certain number, it's harder to, and I definitely, you know, felt a version of that. And that's when

had to, you know, kind of like adapt and change my style of not being able to just like have that level of connection with the entire team, but still be able to have, you know, the confidence in data and whatnot that I needed to be able to make decisions. And, you know, obviously a huge part of that is really just like, you know, finding other ways to get that approximation, but also just like building a fantastic leadership team. And, you know, definitely been very fortunate to be able to build an amazing leadership team over the years. That's actually knock on wood, but a very consistent leadership team in S3 and LinkedIn. So that, you know, that was a huge part of that journey. The other part of your question, you know, what would I do differently?

I think the, you know, how do you frame it? Like if I would, you know, call myself 10 years ago and give some advice, I think if anything, it would be, you know, there's always some set of things that you know you need to do because it's the right thing for the team, you know, there's always some set of, uh, things that, you know, you need to do because it's the right thing for the team, the company, et cetera, but you're concerned about how to get there, or you're concerned about how people are going to react, or like one kind of silly example that I remember was way, way back in the day when I, when I first joined LinkedIn, I mentioned like, you know, this kind of opposite inch thing. And like literally,

um, folks that were developers didn't have shells. They could not access the production environment and that was actually an easy one for me. There were a lot of folks that were concerned like, hey, if we give people this access, what's going to happen? It's like, we'll manage it. It's going to be fine. But another version of that was putting

developers on call and it's like, hey, it's not just going to be the ops folks. You're on call as well. you wrote this code like we need you know we need your help and that you know there was there were you know there were concerns you know there were folks who were concerned about you know everything from like hey that's not necessarily what these people are hired to do this is a big transition etc and you know that's one of those things where it's like there's no question this is the right thing to do but there is a like how do we get there and how do we navigate it and so you know

that's one example there were many you know things like that over the years that i think every single one of them um i just would have done it faster you know and it's always like if you have that conviction that like it's the right thing to do even though you know it's um there are things you're gonna have to figure out along the way, just like lean in and do it and start figuring it out. Um, uh, because when you have that conviction that it's the right thing to do and it's going to have the right outcome, it always does. And it always, you know, leads to the best outcome for the, for the company. Um, and where you're wrong, you tweak it along the way. That's, you know, and where your assumptions are wrong, you iterate. And, um, uh, you're wrong, you tweak it along the way that's, you know, and where your assumptions are wrong, you iterate.

And, um, uh, so I think, you know, that's something that took me a while to get, uh, more comfort with. And it's probably a human nature thing a little bit too, to, you know, be trepidatious about some of those types of situations when, you know, you've got like, you know, you're impacting the, you know, the lives of a bunch of folks and like their happiness at work. Like, obviously it's something you take super seriously. You want to be, you know, you're impacting the, you know, the lives of a bunch of folks and like their happiness at work. Like, obviously, it's something you take super seriously. You want to be careful about. Yeah, move faster with some of those decisions.

To dig in a little on that, when you're making some of these decisions that you think are absolutely right for the company and right for the team. However, what we perceive as right can sometimes be different. So when you're about to make a change like that, and not everyone's on board right away, how do you navigate a situation like that? When you have the conviction that this is the right path forward, but not everyone sees it that way yet? Yeah, I mean, I think it, you just reaching some alignment as to why it's the right outcome for the company. Even if we don't necessarily know how to get there yet, are we in agreement that it's the right thing to do? And if not, let's iterate on that and figure out what we think the right thing to do is. and then you know when it does get into the point of like figuring out how to navigate it um uh you

know we'll go like we're never gonna we're never gonna make a bad we're never gonna intentionally make a bad decision we're never gonna make a decision that is going to you know intentionally you know uh make people unhappy that's going to uh you know it's true about engineering decisions as well we're never going to make a decision that's that's going to have a bad technical outcome like if if we don't like if we in agreement, this is the right place to go. And you're comfortable that, like, even though we don't know how to get there yet, like, we're going to solve it. We aren't going to do it until we figure out that solution. Basically, it's kind of in our control. We can move as quick or slow roll as

much as we want, but let's move towards what we think is the right thing and what we would be proud of building. The only other thing I would add on top of that, too, is just being super transparent with all the data. Everyone should have access to as much know, as much of the data, all the data as to like, this is why we're making the decision.

There's no, you know, this is, this is it. I think as engineers, you know, that, that just kind of disarms us in general when we see like, okay, like I don't necessarily have to agree with, but I understand why and how we made the decision. That makes sense. There's so much to take there. And I feel like there's so much more I can ask, but we're getting towards the end. And we have just a couple more questions for you. So many people at LinkedIn have mentioned this to me. And this is something that even I have

experienced that when we're in a group discussion with you or even a one-on-one you have this incredible ability to put people at ease and this is something I've experienced multiple times in addition to that when you have feedback on something that's been presented to you I feel that it has always been delivered in a way that is meant to make the presentation better or the approach better instead of challenging it. So how do you think about this or these interactions? And is this something that you are very conscious about or intentional about? So, again, a lot in there. I think that, you know, everything, uh, worth doing in life requires like some intention and some practice for sure. And I've learned so much about how to collaborate with and communicate with folks, um, uh, over the years, the, you know, I think some of the underjumps of what I've mentioned is like, for me, you

know, my prioritization is always, you know, the company and then my team and then myself. And I think the, you know, when your priorities are in, you know, some version of that order and you can help and everyone else's priorities is, some version of that order, um, and you can help and everyone else's priorities is in some version of that, or that's when, that's when things, you know, really start to work. And that's when you can, you know, much more easily put yourself in someone else's shoes and really kind of slow yourself down enough to think about like, okay, like where are they coming from and what's their perspective and what am I missing? Because if what I think is totally the right solution to this,

they're not seeing, then I'm either not communicating this right or not seeing what they're, like, they have a perspective that I don't have. Like, I'm missing something, obviously, so what is it? And, I mean, so that for me has definitely been intentional to a certain extent of, like, getting more into that mode because it, like it it feels right like it's like you know that that feels like a great way to communicate and partner with folks um the you know i the other thing i would say too is like just in general i i very much become an optimist um uh the you know the older i get and a younger version of me me would probably be surprised to hear me say that.

And I definitely, many years ago, identified as that kind of cynical engineer that could find flaw in everything or find some weird intent in everything that's probably not there or almost certainly not there. And I think I just reached a point and like through, you know, learning from like some of these fantastic people that I've had a chance to work with and learn from that, you know, that, that, that viewpoint isn't great and, and, you know, it doesn't, it didn't always serve me well. And anyway, where I'm going with this is I think, you know, probably sometimes I think that optimism in, you know, like I, I can bring that out in others. And so maybe, you know, that's some of what you hear from, you know, probably sometimes I think that optimism in, you know, like I, I, I can bring that out in others. And, uh, so maybe, you know, that's some of what you hear from,

you know, people feeling, you know, the, the at ease or, you know, feeling yourself is, um, some culmination of all those, all those things that I just mentioned. Uh, also like just personal connections have always been very important for me. I just, I really, like, I enjoy connecting with and, um, uh with and partnering with people. And I think that, you know, it goes back to like what I mentioned about how I kind of grew the team and my management style

and really building and maintaining connections with a lot of folks. Like, you know, I did that because that felt like it felt natural to me and I enjoyed it and got energy from it. I like that. I like that a lot. I think a personal takeaway for me there when collaborating with people is to think about

what they are seeing that I don't and what perspective do they have that I don't yet. And if they are not seeing my perspective, then it's probably because I am not communicating it right. I think that's a very positive way of collaborating with people. And this reminds me of the quote, and I forget who said it, where it goes something like, seek first to understand and then to be understood.

I think that makes a lot of sense. Yeah, 100%. That I, you know, at this point, I'm convinced it's usually one of those two outcomes. If there is a miscommunication on something, it's either I'm not doing a good job communicating it or they have a perspective that I haven't sought to understand well enough yet. I'm going to ask a very cheeky question to stay on brand. So as you mentioned, I like your style of really trying to get to connect with people. But then you also mentioned that there's kind of a limit in terms of how many people that you can really kind of get to know and then kind of keep in touch. So I guess what are the dimensions when you think about in terms of if you have to sort and filter?

How do you sort? Because to me, I guess the two dimensions I think about uh you know how much joy i get out of that interaction is that sort of the right way and then some kind of weighting of the two um or is there like a more sophisticated framework that i should think about uh we can't cut this part out if it's a little bit too cheeky you know i'm just kind of curious um i know i don't think it's cheeky at all i think it's actually a pretty pretty deep question it's a fantastic question i don, I don't know if I've really ever thought about that in terms of like, is there a framework this sounds even a little self-serving, you know, I will seek out, you know, folks that I get energy from talking to and because like, you know, positivity absolutely breeds positivity. And, and I love, you know, especially on like, you know,

a challenging day when, you know, for whatever reason, you know, a bunch of things have happened in a way that just, you know, have you feel, you know, feeling like you have the weight of the world on you a little bit, like a conversation with someone who you know has that that energy like can just totally you know oney your your your entire mood and perspective so um like absolutely one thing i do in general is you know have that you know a set of folks that um you know, not only I enjoy and I learn from, but also, you know, they bring, you know, I take some of their energy, you know, probably. But that's an important part of how I keep going, for sure. I feel like we can keep going here and ask you a lot more questions. But I do realize that we are at time. This has been an absolute pleasure to spend the time with you, Bruno. We've learned so much. Thank you so much for taking the time and we really appreciate it.

Yep. Likewise. It's been a blast. Thanks for having me. Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com. We would love to hear from you. Until next time, take care.