Screaming in the Cloud - Continuous Integration and Continuous Delivery Made Easy with Rob Zuber

Episode Date: February 12, 2020

About Rob ZuberRob Zuber is a 20-year veteran of software startups; a four-time founder, three-time CTO. Since joining CircleCI, Rob has seen the company through its Series B, Series C, and Series D funding and delivered on product innovation at scale. Rob leads a team of 150+ engineers who are distributed around the globe.Prior to CircleCI, Rob was the CTO and Co-founder of Distiller, a continuous integration and deployment platform for mobile applications acquired by CircleCI in 2014. Before that, he cofounded Copious an online social marketplace. Rob was the CTO and Co-founder of Yoohoot, a technology company that enabled local businesses to connect with nearby consumers, which was acquired by Appconomy in 2011.Links ReferencedTwitter: @z00bLinkedIn URL: https://www.linkedin.com/in/robzuber/Personal site: https://www.crunchbase.com/person/rob-zuber#section-overviewCompany site: www.circleci.com

Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored by in the Cloud. care about, like whether you should sunset that feature that no one uses because it's costing you a fortune and eating away at your margins. Cloud Zero will also alert you right away when you push out a feature that, say, unknowingly doubles your S3 costs before you get hit with that enormous bill. Go to cloudzero.com to kick off a free trial. And my thanks to them for sponsoring this episode. This episode is brought to you by DigitalOcean, the cloud provider that makes it easy for startups to deploy and scale modern web applications with, and this is important around the world love, you can control your cloud

infrastructure costs and have more time for your team to focus on growing your business. See what businesses are building on DigitalOcean and get started for free at do.co slash screaming. That's do.co slash screaming. And my thanks to DigitalOcean for their continuing support of this ridiculous podcast. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Rob Zuber, CTO of CircleCI. Rob, welcome to the show. Thanks. Thanks for having me. It's great to be here. It really is, isn't it? So you've been doing the CTO dance, for lack of a better term, at CircleCI for about five, six years now at this point? Yeah, that's right. I joined five and a

half years ago. I actually came in through an acquisition. We were building a CI CD platform for mobile, iOS specifically, and there were just a few of us. And I came in in an engineering role, but within, I think, a year had taken over the CTO role, and I've been doing that since. So for those of us who've been living under a rock and recording podcasts, SEICD, or continuous integration slash continuous delivery, has gone through a bit of, shall we say, evolution since the term first showed up. I mean, my first exposure to it many moons ago was back when Jenkins was still called Hudson. And it was the box that you ran that it would wait for some event to happen, whether it

was the passing of time, a commit to a particular branch, someone clicked a button, and then it would run a series of scripts, which sort of lent itself to the idea of the Hacker News Anthem. That doesn't look hard. I can build that in a weekend. And now we've seen a bit of growth in that space of not just, I guess, the systems you can run yourselves, but also a lot of the SaaS offerings around this. So that's the, I guess, the moron's journey, from my perspective to path through CICD. That's almost certainly lacking nuance. What is it, I guess, in the real world with adults talking about it? Yeah. So I think it's a good perspective or it's a good description

of the perspective that many people have. Many people enter into this feeling that way. I think specifically when you talk about cloud providers and CircleCI, we do have an on-prem offering or behind the firewall. No one really runs anything on-prem anymore. But we have an offering for that market. But the real leverage is for folks that can use our stuff, multi-tenant SaaS cloud offering. Because ultimately, it's true. Many people start with something simple from a code-based perspective. I'm starting out. I've got a small team.

We have a pretty simple project, maybe a little monolith, Ruby on Rails, something like that. I guess, actually, I think in the time of the start of CircleCI, probably not too many people kick off a Rails monolith these days, because if you're not using Kubernetes and Docker, then you're probably not doing it right. So the Kubernetes and Docker people tell us. Yeah, exactly. They will proudly tell you that. So we'll come back around to that point if we want to. But so you have simple project and you have simple CI, right? Like you may just have a simple script that you're putting in a Jenkins box or something like that. But what ultimately ends up happening is it gets complicated. And as it gets complicated, it becomes a bigger and bigger distraction from the thing that you're really trying to do, right? You're trying to build a business to, I don't know,

to do ride hailing, to do scooter sharing. I mean, what's big these days, right? You might be trying to do any of the- Oh, my side project is Twitter for pets. We're revolutionizing the world of pet communication. Right, and do you want to spend your time working on pet communication or on CICD, right? CICD is a thing that we understand

very well. We spend our time on it every day. We think about some of the depths of it, which we can go into in a second. And one of the things that gets complicated amongst others is just scale, right? So you build a big team, you have multiple projects, and you have that one box under your desk where you said, oh, it's not that hard to build CINCD, right? And now everybody's waiting for their stuff to run because someone else got in there before them. And you're thinking, okay, well, how do I buy, you know, maybe you're not buying more boxes, you're building out something in a cloud provider. And then you're worrying about auto scaling because it starts to cost you too much to run those boxes. And how do you respond to the amount of load that you have on any given day? Because,

you know, you're crunching for a deadline versus everybody's taking a week off um you know and then you want to get your build done as quickly as possible so you start figuring out how to parallelize the work and spread it across those machines uh the list goes on and on and this is this is the reality that everyone runs into as they scale their work and we do that for you so while it seems simple and and you know I said I came in through an acquisition we were building CI and CD for iOS and I was that person I said this seems really simple we should build it and put it in the market and it didn't take us very long to get that that first version to build and it had to be generic to support many different types of

customers and their particular builds but it was a it was a small start but the you know we we started to run into the same problems and then of course as a business we ran into the problem of getting access to customers and all those things. And that's why we joined CircleCI. And sort of that became what is now our iOS offering. But there is a lot of value that you can get quickly, you know, to your point. But then you start focusing time and energy on that. Right.

I often refer to it, others in the industry refer to these sorts of things as undifferentiated heavy lifting, right? Something that becomes big and complex over time and is not the core of your business. And so then as you start to invest in it, as we invest in it, then we build capabilities that most people wouldn't bother to build when they write that first bash script off a trigger or whatever around helping you get your project set up, handling the connection into hooks, handling authentication so that different users only have access to the code they should have access to, maybe isolating access to production secrets, for example, if you're doing deploy, the kinds of things that keep coming up over and over in CI and CD that people

don't think about on that first pass, but end up hunting them down the road. What do you think that people tend to misunderstand the most about CI, CD, as you take a look at that throughout the ecosystem? I mean, from my perspective, when it was a box that you ran in behind the firewall, as you say, the problem was is that everyone talked about, oh, yes, we use cattle, not pets, except the box that does the builds. And of course, that box has a bunch of hand-built stuff on it that's impossible to replicate. It has extraordinary permissions into production environments and can do horrifying things. And it was always the star of various security finding reports. So there are a number

of us who came up from an operations side viewing CICD as in some ways a liability, which I understand is a very biased and one-sided perspective. But going beyond that, what are people missing? What are they not seeing about the CICD landscape? One thing that I think is really interesting there, like, well, one thing you call that was just resiliency, right? So we think about that in the way that we operate that system. And so, you know, we have a world of cattle because we've managed to think about that as a true offering, right? So as you scale and you start to think, oh, how do I make this resilient inside my operation? That's going to become a challenge that you face. The other thing that I

think about that I've noticed over the years is I want to call it division of labor or division of responsibilities. So many of those single instance or even multi-instance self-managed CICD tools end up in a place where, you know, past any size of team, honestly, somebody needs to own it and manage it to make sure it's stable. And the changes that you want to make as a developer are often tied to being basically being managed by that administrator so to be a little clearer if I have a group responsible for running CI and CD then and I want to start building a different type of code or a different project and it requires a plugin or an extension to the CI and CD platform or CI CD tool then I need to probably file a ticket and wait for another department who is generally not super motivated in me you know to get my code out into production

to go make a change that they are going to evaluate and review and decide what or maybe creates conflict with something somebody else is doing on that system. And then you say, oh, well, actually, we can't have these co-installed. So now we need two systems. And it's that division of responsibilities. Whereas having built a multi-tenant cloud offering, we could never have that, right? There's no world in which our customers say to us, hey, we want this plugin installed. Can you go do that for us, right? So everything that is about how the development team

thinks about their software and how they want their build to run, how they want their deploys to run, et cetera, needs to be in the hands of the developers. And everything that is about maintenance and operation and scale needs to be in our hands. So it's created a very clear separation out of necessity, but one that even, you know, I mentioned that you can deploy CircleCI yourself and run it within a team. And in large organizations, that separation really helps them get leverage. Does that make sense?

It really does. And I think we're also seeing a change in perspective around resiliency and how this works. I once worked at a company, I will not name, where they were, it was either CircleCI or TeamCity. This was years and years ago where I don't recall exactly what they were using, but it doesn't matter because at one point the service took an outage and in typical knee-jerk reaction well that can never happen again so they wound up doing all of the CI CD work for some godforsaken reason on a Raspberry Pi that some developer brought in and left in the corner of the office and surprise it took an awfully long time for tests

to run on basically an underpowered toy project and And the answer there was to just use less tests because you generally don't need to run nearly as many. And I just stared at people for the longest time when it came to that. I think that one of the problems that we still see, I know when I write code myself, I'm as guilty of this as anyone, I'm a terrible developer and don't believe in tests.

So the CICD pipeline that I tend to look at is more or less a glorified script runner. Whenever I make a commit to this branch, go ahead and run the following three-line script that does a serverless deployment and puts it where it needs to go, and then I'll test it manually. Or it's a pre-production environment, so it's not that big of a deal. And that can work for some use cases, but it's also a great thing that no one actually depends on the stuff that I write for day-to-day business operations or anything critical. At what point does it stop being a script runner?

Well, to the point of the scale, I think there's a couple things that you brought up in there that are interesting to me. One is the culture of testing. And it feels like one of these sort of areas of software development, because I was around in a time when no one really understood what it was to do automated testing. I won't even go into TDD, but just in general, why would I do that? We have this QA team. It's so much, you know, it's cost effective to give it to a bunch of people. Or I mean, thinking backwards or thinking back on that, it all seems a little bit, well, wrong. But getting to the point where you've worked effectively with tests takes a little bit of effort.

But once you have that, once you've sat and worked on something and had the feedback loop of, oh, this thing's not working. Oh, I'll just change this. Now it's working. Like really having that locally as a developer is super rewarding in my mind. And so, and enabling, I guess I would say as well. And so then you get to this place where you're excited about building tests, right? Especially as you're working in a team. And then

culturally you end up in a place where, you know, I put up a PR and someone else looks at it and says, I see you're making an assumption or I believe you're making an assumption here, but I don't see any way that that's being validated. So please add testing to ensure that is actually true. Both because I wanna make sure it's true now, but when we both forget that you ever wrote this and someone else makes a change,

your assumptions hold. Or someone can understand that you were making those assumptions and they can make appropriate changes to deal with it. And I think as you work in a team that's growing and scaling and beyond your sort of pet project, once you've witnessed the value of that, you don't want to go back. And so people do end up writing more and more tests. And that's what drives the scale, at least on the testing and CI side, in a way that you need to then manage that, right? And going the opposite direction of what you're describing, which is, hey, let's just write fewer tests and use cheaper machines. People are recognizing the value and saying, okay, we want that value, but we don't want to bottleneck everyone with an hour long build, you know, to run all these. So how do we get a system that's going to scale and support that? And that's, what's fascinating is watching that start

to percolate beyond the traditional web applications with particular blessed languages and into other things. For example, in my copious spare time, I'm the community lead for the Open Guide to AWS, which is a GitHub project that has 25,000 stars or so, so you know it's good, where it's just a giant markdown document that lists the 10,000 tips and tricks that we all wish we'd known when we'd gotten started with AWS and in a format that's easily consumable. The CICD approach we have right now, which I believe is done through Travis, is it just winds up running a giant link checker

in parallel across the thousands of links that are, sorry, I want to say 1,200 links that are included within that document. There's really not a lot else we can do in that type of environment. I mean, a spell checker with all of the terms of art involved would more or less segfault itself to death as soon as it took a look. But other than making sure we don't have

dead links in, it feels like there's not a lot of automation or testing opportunity in something like that. Is that accurate? Am I completely wrong and missing something? I've never built that particular site. So I mean, it sounds reasonable. I think that going the other way, we often think about, you know, before we kick off a large, complex set of testing for a more complex application, maybe than a Markdown document, a lot of people now will use things similar to what you're using. Like maybe I, part of my, part of my application is a bunch of links to outside docs or outside sites that I'm referencing. Or, you know, if I run into a problem, I link you to our help site or something and making sure all that stuff is validated, doing linting on the structure and format of code itself. And one of the things that, that comes up as you, as you scale out of, you know, the individual script runner is doing that work in parallel. Right. So I can say, you know what, do the linting over here, do the link checking over here, only use very small boxes for those.

We don't we don't happen to have Raspberry Pis in our infrastructure, but we can give you a much smaller resource, which costs you less if you're not going to be pushing the limits of that. But then if you have, you know, big integration tests or something which need more space, then we can provide that as well, both in a single channel or pathway to give you the room to move faster and then to break that out and run it you know break up your work like um at an extreme example and of course anyone who's done parallelization knows there's there's cost to splitting up work in uh like the management overhead but if you have 1200 links like you could check them all at the same time, right? I doubt that would be a good use of our platform.

But you could check 601 and 600 and another, or 300s at a time or whatever, and find the optimal path if you really cared about getting that done more quickly. Right. Usually, it's not that big of a concern. And usually, it winds up throwing errors on existing bad links, not something that has been included in the pull request in question. So again, there's nothing that is so awesome that I can't horribly misuse it for something ridiculous. It's my entire stock and trade. It's why I believe Route 53 remains the best database option for everyone. But it's fun going through this space and just seeing how things have evolved. One question I do have, since you come from a background by way of acquisition that was

aimed squarely at this, historically, it seems that running a lot of testing on mobile devices, specifically iOS devices, was the stuff of nightmares because you couldn't really run that in any meaningful way in a virtualized environment. So it generally required an awful lot of devices. Is that still the case? Has that environment changed radically since I last worked at a mobile shop? I don't think so, but I think we've all started to think a little bit differently. And so we got started in that business because we were building iOS apps and thought, wow, the tooling here is really frustrating. And to be clear, at CircleCI and at that business, we were solving the problem of managing the machines themselves.

So the portion of the testing that you would run effectively in a simulator, not the problem of the device farm, if you will. But one of the things that I remember, and so this is maybe 2014, late 2013, early 2014, as I was working on mobile apps, was people shifting the MVC layers a little bit such that the thing that you needed to test on a device was getting smaller and smaller, meaning putting more logic in, I forget what the name was specifically, but it was like the view,

I don't want to try to even guess, but basically pulling logic out of the actual rendering and down into what we'll call state transitions, I guess. And if you think about that in modern day and look at maybe web frameworks like React, you're trying to just respond with rendering on top of a lot of state change that happens underneath that.

And in that model, if you thin out the user interface portion, you make a lot more of your code testable, if that makes sense. So the reason we're all trying to test on all these different devices is often that we've baked a lot of business logic into the view layer does that make sense and yeah it absolutely

does and so please continue oh and so in order to instead of saying well all our logic's in the view layer so let's get really good at testing the view layer which means massive device farms and a bunch of people testing all these things let's make that layer as thin as possible and there's there's analogies for this in you know even how we do service design these days and structure the architecture of systems basically make the boundaries as thin as possible and the interaction with the outside world as thin as possible and that interaction with the outside world as thin as possible. And that gives you much more capability to effectively test the majority or much larger portions of your business logic. So the device farm problem is still a problem. People still want to see how something specifically renders on a particular screen or whatever whatever um but by minimizing that the amount that you have to invest in that gets smaller so as you you mentioned device farm which is app choice

given that that is the name of an aws service that has a crap ton of mobile devices that you can log into and it's one of my top candidates for the did i make this service up to mess with you competitions it does lead us to an interesting question. CICD has gotten an increased amount of attention lately from pretty much everyone. And AWS, as is typical for Amazon, tends to lie awake at night worrying that someone somehow is making money that isn't them. So their product strategy distills down to yes. So they wound up releasing a whole bunch of CICD-oriented products that at launch were, to be polite, terrible. And over time, they've gotten slightly better, but it's still a very confusing ecosystem there. And then we see things like

Azure DevOps, who it seems is aimed at a very similar type of problem, and they're also trying to challenge Amazon on the grounds of terrible names of services. But we're now seeing an increased focus from the first party providers themselves around the CI, CD space. What does that mean for existing entrenched players who have been making a specialty out of this for a lot longer than these folks have been playing with?

Yeah, so it's a great question. There's, I think about the approaches very Yeah, so it's a great question. I think about the approaches very differently, which is probably unsurprising. I mean, speaking of lying awake at night or spending all day thinking about these things, like this is what we do. So the thing, and you've used the term script runner a few times in the conversation. The thing that I see when I see someone like AWS looking at this problem is basically, you know, people are using the way that I think about it as maybe less the money, although it translates pretty quickly. People are using compute to do something. You know,

can we get them to do that with us? And, you know, oddly enough, a massive chunk of CircleCI runs on AWS, so it doesn't really matter to them one way or another. But they're effectively looking to drive compute hours and looking to drive, you know, a pathway onto their platform. And so one thing about that is it doesn't really matter to them, in my perspective, whether people use that particular product or not. And as a result, it gets the product investment that you put in when that's the case. And so it's a sort of a check the box approach. Like, hey, we have CI and we have CD like other people do. Whereas when we look at CI and CD, we've been talking about some of the factors like scaling it effectively and making it really easy for you to understand what's going on.

We think about very much the core use case. What is one of our customers or users doing when they show up? How do we do that in a way that maximizes their flow, minimizes the overhead to them of using our system, whether it's getting set up and running really quickly, like talk about being in the center of how much of the world is developing software. So we see patterns, we see mistakes that people are making and can use that to inform both how our product works

and inform you directly as a user, hey, I see that you're trying to do this, you know, it would go see that you're trying to do this, you know, it would go better if you did this. And so I think both from the, honestly, the years that we've been doing this and the amount that we've witnessed in terms of what works well for customers, what doesn't, what we see going through just from a data perspective, you know, as we see going through just from a data perspective, as we see hundreds of thousands of builds running, that rich perspective is unique to us because, as you said, we're a player that's been doing this for a really long time

and very focused on it. And we treat the experience with, I guess, I'm trying to figure out a way to say this that doesn't sound as bad as it might. But a lot of people have suffered a lot with CI and CD, right? There's a lot that goes into getting CI and CD to work effectively and getting it to work reliably over time as your system is constantly changing. And honestly, there's a lot of frustration and we come in to work every day thinking about minimizing that frustration so that our customers can go spend their time doing what matters to them. And again, when I think you, you know, you sort of, a lot of these big players present you with a runtime in which you can execute a script of your choosing. It's not, it's not thinking about the problem in that way.

And I don't see them changing their perspective. And so I don't really, honestly, I just don't worry about them. Which is a very fair tack to take. And it's interesting watching companies and as far as how much time and energy they spend worrying about competition versus how much they focus instead on customers. So to turn it around slightly, what makes what you do challenging in some respects, I would imagine, is that a lot of your target market is themselves developers. Developers, in my experience, are challenging customers in that, first, they tend to devalue their own time to the point where, oh, that doesn't sound hard. I'll build that overnight.

Secondly, once you finally win them over to the idea of paying for something, it's challenging to get them to have the necessary signing authority. So at best, they become champions. But what you do has to start with developers in order to win widespread adoption and technical buy-in. How does that wind up manifesting as approach to, well, some people call it developer relations, developer advocacy. I refer to those folks as dev relippers because I have problems. But how do you folks view that? Yeah, it's a really insightful view, actually, because we do end up in most of our customers or in the environments of our customers, however you want to describe it, as a result of the enthusiasm of individual developers, development teams, much more so than, you know,

there are many products in certainly an enterprise software, and I don't really think purely in enterprise, but there are many products that can only be purchased by the CIO or the CTO or whatever, right? And so we do, I mean, to your question of developer relations, we spend a lot of time out in the market, you know, talking to individuals, talking at conferences, writing content about how we think about the space and things that people can do but we are a very product driven company um meaning both that's that's what we think about

first and then support it with these other things but second um we we win on product right like we don't win in the market because you thought a blog post that we wrote was really cool. That might make you aware of us. But if you don't love the product, I mean, developers, to your point, they want to use things that they really enjoy using. And when developers use the product and love the product, then they champion it and they get access because they might work on, you know, a side project or an open source project, or maybe they worked in another company that used CircleCI and then they go somewhere else and they say, what are we doing? Life is so much better if we use CircleCI, those sorts of things. But it very much comes from the bottom up. It's pretty difficult to go into an organization

and say, hey, you should push this down to all of your developers. There's a lot of rejection that comes from developers on mandated tooling. And so we have to provide knowledge. We have to provide capabilities in our product that appeal to those other folks. So for example, administrators of our tooling

or when it gets to the point where someone owns, you know, the owns how you use CircleCI versus just being a regular user of the product, we have capabilities to support them around understanding what's happening, around creating shared capabilities that multiple teams can use, those sorts of things. But ultimately, we have to lead with product.

We have to get into the hearts and minds of the developers themselves and then grow from there. And everything we do from marketing, developer relations, myself, I spend a lot of time talking to customers who are out in the market, is all about propping up or helping people, helping raise awareness effectively. But there's nothing that we can do if the product doesn't meet the needs of our customers. And that's what it seems like it comes down to a fair bit. It's always weird to consider that at its heart, developer relations is marketing. And the folks I talk to who argue

against that, it seems that it comes from a misunderstanding of what marketing actually is. It's not buying ads in airports. It's not doing podcast advertisements, which is a subject near and dear to my heart. It's not about annoying people by showing up at their office with a sales team. It's about understanding what their challenges and problems are and then positioning a solution that ideally solves them in a place that, and in a way that is, that they can be receptive to. Instead, people tend to equate marketing to this whole ridiculous statistics-driven nonsense that doesn't really resonate with anyone, and I think that that's unfair to everyone involved. That said, I will say that having

spent a fair bit of time in this space, I've yet to see anything from CircleCI that has annoyed me to the point where I would have remembered it, which is awesome. I don't see it in in-flight magazines generally. I don't see it on obnoxious people trying to tackle me as I walk through an expo hall and want to scan my badge. It just seems very well executed and you have some very talented people working for you. To that end, you are largely a distributed company, which is fascinating. Did it start that way? Did it happen that way by a quirk of fate? Yeah, those two things probably come together. So the company from very early days, now I wasn't there, but I think some of our earliest engineers were distributed. And, you know, the company started out basically entirely as engineers. It's a team solving problems of other engineers, which is a, it's a fun challenge.

And so there were there were early participants who were distributed, mostly, you know, when you when you start a company, and no one has ever heard of you, and no one knows if you're going to be successful. Going and recruiting is generally a different game than when you're certainly when you're where we are now. And so you know, there were some personal relations that just happened to connect with people around the globe who wanted to participate. And so we we started out pretty early with some distribution. And that led to structuring the org in a way, you know, both from a tooling and process perspective, a lot of that sort of happens organically, but building a

culture that really supported that. You know, I personally am based in the Bay Area, so we have a headquarters in San Francisco, but it doesn't really make a difference if I go in versus, you know, just stay and work from home on any given day because the company operates in such a way that, that, that distribution is, is completely normal. We accidentally did the same thing. My business partner and I used to live across the street from each other and we decided to merge a week before he moved out of state to Portland. So awesome.

Great. We have wonderful timing on all of these things. It's fun to build that way from the ground up. The challenge I've always seen is when you start off with having a centralized office and everyone's there except this one person who, no matter how you try to work around it, is never as involved. So it feels like the sort of thing you've absolutely got to be building from day one, or otherwise you're going to have a massive cultural growing pain as you try to get there. Yeah, I think that's true. So I've actually been that one person. I, at some point in my career prior to CircleCI, was helping out a company founded by some friends of mine based in Toronto. I grew up

in Toronto and I kicked off a project. And then the project grew and grew until I was the one person out of maybe 50 or 60 who wasn't in an office in Toronto. And it got to the point where no one remembered who I was. And I was like, cool, I think I'm done. I'm out. I was fine with that. It was always meant to be a temporary thing, but I really felt that transition for the organization. And I would say in terms of growing, I mean, yes, if you start out, it kind of goes both ways. If you start out distributed, you're going to remain distributed. There are certain things that get more challenging at scale, right?

If everybody is sort of just in their home all over the globe, then the communication overhead continues to increase and increase and just understanding who people are, who you should be talking to. You need to focus. There's always a time zone hierarchy too. Oh, the time zones are a delight. Yes. And I would say like we talk a lot about

in this industry Dunbar's number and sizes of teams and the points at which things get more complex. And I think there's probably a different scale for distributed teams. It takes fewer people to reach a point where communication gets challenging. And trust and all the other things that go with sort of Dunbar's views.

And then the, so it's, you know, you kind of have that challenge and then you start to think, oh, well, you know, then you have some offices because we actually have maybe six physical offices, partly because in our go-to-market org, we've started to expand globally and put people in regional offices. So there's kind of this interesting disconnect. I don't know about disconnect, but there's a split in how we operate in different parts

of the org. And I think what I've seen people, well, I don't know about succeed, but I've sort of seen people try when you start out with one org or sorry, one location is let's not jump to like that one person somewhere else and then one person somewhere else kind of thing, but build out a second office, build out another office, like pick another location where you think you, it's often certainly where we are in the bay area um it's often driven by just this market right finding finding talent finding uh people who want to join you hanging on to those people when there are so many other opportunities around um is tends to be much more challenging and when you offer people alternatives like you

can you can stay where you are but have access to a cool and interesting company or you can work from home which a lot of people value then there's different things that you bring to the table so I see a lot of people trying to expand in that way but when you are so office centric a second office I think is a smoother transition point than just suddenly distributing people because especially the first and second one unless you're hiring in a massive wave are really going to struggle in that environment. I think that's probably one of the more astute things that's been noticed on this show in the last couple of years. If people want to hear more

about what you have to say and how you think about the world, where can they find you? I would say on our blog, I tend to write stuff there as do other people. There's a lot of, you talked about having great people in the organization. We have a lot of great people talking about how we think about engineering, how we think about both engineering teams and culture, and then some of the problems we're trying to solve. So off our site, circleci.com, go to our blog. And then I tend to speak and hang out on podcasts and do guest writing. I think I'm pretty easy to find.

You can find me on Twitter. My handle is ZubZ00B. I'm not super prolific, but if someone was to track me down and ask me something, I'd probably be more than happy to answer. And you can expect some engagement as soon as this goes out. Thank you so much for taking the time to speak with me today. I appreciate it. Yeah, thanks for having me. This was a ton of fun. Rob Zuber, CTO at CircleCI. I'm Corey Quinn, and this is Screaming in the Cloud.

If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. If you've hated this podcast, please leave a five-star review on Apple Podcasts, along with something amusing for me to read later while I'm crying. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold. this has been a humble pod production

stay humble