Screaming in the Cloud - Holiday Replay Edition - Continuous Integration and Continuous Delivery Made Easy with Rob Zuber

Episode Date: December 22, 2022

About RobRob Zuber is a 20-year veteran of software startups; a four-time founder, three-time CTO. Since joining CircleCI, Rob has seen the company through its Series B, Series C, and Series D funding and delivered on product innovation at scale. Rob leads a team of 150+ engineers who are distributed around the globe.Prior to CircleCI, Rob was the CTO and Co-founder of Distiller, a continuous integration and deployment platform for mobile applications acquired by CircleCI in 2014. Before that, he cofounded Copious an online social marketplace. Rob was the CTO and Co-founder of Yoohoot, a technology company that enabled local businesses to connect with nearby consumers, which was acquired by Appconomy in 2011.Links:Twitter: @z00bLinkedIn URL: https://www.linkedin.com/in/robzuber/Personal site: https://www.crunchbase.com/person/rob-zuber#section-overviewCompany site: www.circleci.com

Hello and welcome to a special holiday replay edition of Screaming in the Cloud. This special edition features one of our favorite conversations from the past few years as we prepare for a new year of shenanigans in 2023. Thanks for listening and happy holidays. Hello and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

If you asked me to rank which cloud provider is the best developer experience, I'd be hard-pressed to choose a platform that isn't Google Cloud. Their developer experience is unparalleled in the early stages of building something great. That translates directly into velocity. Try it yourself with the Google for Startups cloud program over at cloud.google.com slash startup. It'll give you up to a hundred grand a year for

each of the first two years in Google cloud credits for companies that range from bootstraps all the way on up to series A. Go build, and then tell me about it. My thanks to Google Cloud for sponsoring this ridiculous podcast. This episode is brought to us in part by our friends at Pinecone. They believe that all anyone really wants is to be understood, and that includes your users. AI models combined with the Pinecone Vector Database let your applications understand and act on what your users want without making them spell it out. Make your search application find results by meaning instead of just keywords. Your personalization system make picks based on relevance instead of just tags.

And your security applications match threats by resemblance instead of just regular expressions. Pinecone provides the cloud infrastructure that makes this easy, fast, and scalable. Thanks to my friends at Pinecone for sponsoring this episode. Visit pinecone.io to understand more. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Rob Zuber, CTO of CircleCI. Rob, welcome to the show.

Thanks. Thanks for having me. It's great to be here. It really is, isn't it? So you've been doing the CTO dance, for lack of a better term, at CircleCI for about five, six years now at this point? Yeah, that's right. I joined five and a half years ago. I actually came in through an acquisition. We were building a CI CD platform for mobile iOS specifically. And there were just a few of us. And I came in, in, in an engineering role, but within, I think a year had, had taken over the CTO role and I've been doing that since.

So for those of us who've been living under a rock and recording podcasts, CI CD or continuous integration slash continuous delivery, has gone through a bit of, shall we say, evolution since the term first showed up. I mean, my first exposure to it many moons ago was back when Jenkins was still called Hudson. And it was the box that you ran that it would wait for some event to happen, whether it was the passing of time, a commit to a particular branch, someone clicked a button, and then it would run a series of scripts, which sort of lent itself to the idea of the Hacker News Anthem. That doesn't look hard. I can build that in a weekend. And now we've seen a bit of growth in that space of not just, I guess, the systems you can run yourselves,

but also a lot of the SaaS offerings around this. So that's the, I guess, the systems you can run yourselves, but also a lot of the SaaS offerings around this. So that's the, I guess, the moron's journey from my perspective, path through CICD. That's almost certainly lacking nuance. What is it, I guess, in the real world with adults talking about it? Yeah. So I think it's a good perspective or it's a good description of the perspective that many people have. Many people enter into this feeling that way. I think specifically when you talk about cloud providers and CircleCI, we do have a on-prem offering or behind the firewall. No one really runs anything on-prem anymore, but we have an offering for that market, but the real leverage is for folks that can use our stuff,

you know, multi-tenant SaaS cloud offering, because ultimately it's true. Many people start with something simple from a code-based perspective, right? I'm starting out, I've got a small team, we have a pretty simple project, maybe a little monolith, you know, Ruby on Rails, something like that. I guess, actually, I think in the time of the start of CircleCI, probably not too many people kick off a Rails monolith these days because if you're not using Kubernetes and Docker, then you're probably not doing it right. So the Kubernetes and Docker people tell us. Yeah, exactly. They will proudly tell you that. So we'll come back around to that point if we want to. But so you have a simple project and you have simple CI, right? Like you, you may

just have a simple script that you're putting in a Jenkins box or something like that. But what ultimately ends up happening is it gets complicated. And as it gets complicated, it becomes a bigger and bigger distraction from the thing that you're really trying to do, right? You're trying to build a business to, um, I don't know, to do ride hailing, to do scooter sharing. I mean, what's big these days, right? You might be trying to do any of these. My side project is Twitter for pets. We're revolutionizing the world of pet communication.

Right. And do you want to spend your time working on pet communication or on CICD, right? CICD is a thing that we understand very well. We spend our time on it every day. We think about some of the depths of it, which we can go into in a second. One of the things that gets complicated amongst others is just scale, right? So you build a big team, you have multiple projects, and you have that, you know, one box under your desk where you said, oh, it's not that hard to build CI and CD, right?

And now everybody's waiting for their stuff to run because someone else got in there before them. And you're thinking, okay, well, how do I buy, you know, maybe you're not buying more boxes, you're building out something in a cloud provider. And then you're worrying about auto scaling because it starts to cost you too much to run those boxes. And how do you respond to the amount of load that you have on any given day? Because, you know, you're crunching for a deadline versus everybody's taking a week off. And then you want to get your build done as quickly as possible. So you start figuring out how to parallelize the work and spread it across those machines.

The list goes on and on. And this is the reality that everyone runs into as they scale their work. And we do that for you. So while it seems simple, and I said I came in through an acquisition, we were building CI and CD for iOS. And I was that person. I said, this seems really simple. And, you know, I said I came in through an acquisition. We were building CI and CD for iOS. And I was that person. I said, this seems really simple. We should build it and put it in the market. And it didn't take us very long to get that that first version to build. And it had to be generic to support many different types of customers and their particular builds.

It was a small start, but we started to run into the same problems. And then, of course, as a business, we ran into the problem of getting access to customers and all those things. And that's why we joined CircleCI and sort of that became what is now our iOS offering. But there is a lot of value that you can get quickly, you know, to your point, but then you start focusing time and energy on that. I often refer to it, others in the industry refer to these sorts of things as undifferentiated heavy lifting, right? Something that becomes big and complex over time and is not the core of your business. Then as you start to invest in it, as we invest in it, then we build capabilities that most

people wouldn't bother to build when they write that first bash script, you know, off a trigger or whatever around helping you get your projects set up, handling the connection into hooks, handling authentication so that different users only have access to the code they should have access to, maybe isolating access to production secrets, for example, if you're doing deploy, the kinds of things that keep coming up over and over in CI and CD that people don't think about, you know, on that first pass, but end up hunting them down the road. What do you think that people tend to misunderstand the most about CI CD as you take a look at that throughout the ecosystem? I mean,

from my perspective, when it was a box that you ran behind the firewall, as you say, the problem was, is that everyone talked about,, yes, we use cattle, not pets, except the box that does the builds. And, of course, that box has a bunch of hand-built stuff on it that's impossible to replicate. It has extraordinary permissions into production environments and can do horrifying things. And it was always the star of various security finding reports.

So there are a number of us who came up from an operations side viewing CICD as in some ways a liability, which I understand is a very biased and one-sided perspective. But going beyond that, what are people missing? What are they not seeing about the CICD landscape?

One thing that I think is really interesting there, like, well, one thing you called out was just resiliency, right? So we think about that in the way that we operate that system. And so we have a world of cattle because we've managed to think about that as a true offering, right? So as you scale and you start to think, oh, how do I make this resilient inside my operation? That's going to become a challenge that you face. The other thing that I think about that I've noticed over the years is I want to call it division of labor or division of responsibilities.

Many of those single instance or even multi-instance self-managed CICD tools end up in a place where, you know, past any size of team, honestly, somebody needs to own it and manage it to make sure it's stable. And the changes that you want to make as a developer are often tied to basically being managed by that administrator. So to be a little clearer, if I have a group responsible for running CI and CD, and I want to start building a different type of code or a different project and it requires a plugin or an extension to the CI and CD platform or CI CD tool then I need to probably file a ticket and wait for another department who is generally not super motivated and you know to get my code out into production to go make a change that they are going to evaluate and review and decide or maybe creates conflict with something somebody else is doing on that system. And then

you say, oh, well, actually, we can't have these co-installed. So now we need two systems. And it's that division of responsibilities. Whereas having built a multi-tenant cloud offering, we could never have that, right? There's no world in which our customers say to us, hey, we want this plugin installed. Can you go do that for us, right? So everything that is about how the development team thinks about their software and how they want their build to run, how they want their deploys to run, et cetera, needs to be in the hands of the developers. And everything that is about maintenance and operation and scale needs to be in our hands. So it's created a very clear separation out of necessity, but one that even, you know, I mentioned that we, you can deploy CircleCI yourself and run it within a team and

in large organizations, that separation really helps them get leverage. Does that make sense? It really does. And I think we're also seeing a change in perspective around resiliency and how this works. I once worked at a company, I will not name, where they were, it was either CircleCI or TeamCity. This was years and years ago where I don't recall exactly what they were using, but it doesn't matter because at one point the service took an outage and in typical knee-jerk reaction well that can never happen again so they wound up doing all of the cicd work for some godforsaken reason on a raspberry pi that some developer brought in and left in the corner of the office and surprise it took an awfully long time for tests to run on basically an underpowered toy project. And the answer there was to just use less

tests because you generally don't need to run nearly as many. And I just stared at people for the longest time when it came to that. I think that one of the problems that we still see, I know when I write code myself, I'm as guilty of this as anyone. I'm a terrible developer and don't believe in tests. So the CIC CD pipeline that I tend to look at is more or less a glorified script runner. Whenever I make a commit to this branch, go ahead and run the following three line script that does a serverless deployment and puts it where it needs to go. And then I'll test it manually or it's a pre-production environment. So it's not that big of a deal and that can work for some use cases, but it's also a great thing

that no one actually depends on the stuff that I write for day-to-day business operations or anything critical. At what point does it stop being a script runner? Well, to the point of the scale, I think there's a couple things that you brought up in there that are interesting to me. One is the culture of testing. these sort of areas of software development, because I was around in a time when no one really understood what it was to do automated testing. I won't even go into TDD, but just in general, why would I do that? We have this QA team. It's so much, you know, it's cost effective to give it to a bunch of people. Or I mean, thinking backwards or thinking back on that, it all seems a little bit, well, wrong. But getting to the point where you've worked effectively with tests takes a little bit

of effort. But once you have that, once you've sat and worked on something and had the feedback loop of, oh, this thing's not working, I'll just change this. Now it's working. Like really having that locally as a developer is super rewarding in my mind and enabling, I guess I would say as well. And so then you get to this place where you're excited about building tests, right? Especially as you're working in a team.

And then culturally you end up in a place where I put up a PR and someone else looks at it and says, I see you're making an assumption or I believe you're making an assumption here, but I don't see any way that that's being validated. So please add testing to, you know, to ensure that is actually true, both because I want to make sure it's true now. But when we both forget that you ever wrote this and someone else makes a change, your, you know, your assumptions hold or someone can understand that you were making those assumptions and they can make appropriate changes to deal with it. Right. And I think as you work in a team that's growing and scaling and beyond your sort of pet project, once you've witnessed the value of that, you don't want to go back. And so people do end up writing more and more tests.

And that's what drives the scale, at least on the testing and CI side in a way that you need to then manage that, right? And going the opposite direction of what you're describing, which is, hey, let's just write fewer tests and use cheaper machines. People are recognizing the value and saying, okay, we want that value, but we don't want to bottleneck everyone with an hour-long build, you know, to run all these. So how do we get a system that's going to scale and support that? And that's what's fascinating is watching that start to percolate beyond the traditional web applications with particular blessed languages and into other things. For example, in my copious spare time, I'm the community lead for the Open Guide to AWS, which is a GitHub project that has 25,000 stars or so. So, you know, it's good where it's just a giant markdown document that lists the

10,000 tips and tricks that we all wish we'd known when we'd gotten started with AWS and in a format that's easily consumable. The CICD approach we have right now, which I believe is done through Travis, is it just winds up running a giant link checker in parallel across the thousands of links that are, sorry, I want to say 1200 links that are included within that document. There's really not a lot else we can do in that type of environment. I mean, a spell checker with all of the terms of art involved would more or less segfault itself to death as soon as it took a look. But other than making sure we don't have dead links in, it feels like there's not a lot of automation or testing opportunity in something

like that. Is that accurate? Am I completely wrong and missing something? I've never built that particular site. So it, I mean, it sounds reasonable. I think that going the other way, we often think about, you know, before we kick off a large complex set of testing for a more complex application, maybe than a Markdown document, a lot of people now will use things similar to what you're using. Like maybe I'm part of my application is a bunch of links to outside docs or outside sites that I'm referencing. Or, you know, if I run into a problem, I link you to our help site or something and making sure all that stuff is validated, doing linting on the structure and format of code itself. One of the things that

comes up as you scale out of, you know, the individual script runner is doing that work in parallel, right? So I can say, you know what, do the linting over here, do the link checking over here, only use very small boxes for those. We don't happen to have Raspberry Pis in our infrastructure, but we can give you a much smaller resource, which costs you less if you're not going to be pushing the limits of that. But then if you have, you know, big integration tests or something which need more space, then we can provide that as well, both in a single channel or pathway to give you the room to move faster and then to break that out and run, you know break up your work like at an extreme example and of course anyone who's done parallelization knows there's there's cost to splitting up work in

like the management overhead but if you have 1,200 links like you could check them all at the same time right I doubt that would be a good use of of our platform but you could check 601 and 600 and another, you know, or 300s at a time or whatever in find the optimal path if you really cared about getting that done more quickly. Right. Usually it's not that big of a concern, and usually it winds up throwing errors on existing bad links, not something that has been included in the pull request in question. So again, there's nothing that is so awesome that I can't horribly misuse it for something

ridiculous. It's my entire stock and trade. It's why I believe Route 53 remains the best database option for everyone. But it's fun going through this space and just seeing how things have evolved. One question I do have, since you come from a background by way of acquisition that was aimed squarely at this, historically, it seems that running a lot of testing on mobile devices, specifically iOS devices, was the stuff of nightmares because you couldn't really run that in any meaningful way in a virtualized environment. So it generally required an awful lot of devices. Is that still the case? Has that environment changed radically since I last worked at a mobile shop? I don't think so, but I think we've all started to think a little bit differently. And so we got started in that business because we were building iOS apps and thought, wow,

the tooling here is really frustrating. To be clear, at CircleCI and at that business, we were solving the problem of managing the machines themselves. So the portion of the testing that you would run effectively in a simulator, not the problem of the device farm, if people shifting the MVC layers a little bit such that the thing that you needed to test on a device was getting smaller and smaller, meaning putting more logic in, I forget what the name was specifically, but it was like the view. I don't, I don't want to try to even guess, but basically pulling logic out of the actual rendering and down into what we'll call state transitions, I guess. If you think about that in modern day and look at maybe web frameworks like React you know you're you're trying to just respond with rendering on top of a lot of state change that happens underneath that

and in that model if you thin out the user interface portion you make a lot more of your code testable if that makes sense so the reason we're all trying to test on all these different devices is often that we've baked a lot of business logic into the view layer. Does that make sense? Yeah, it absolutely does. Please continue. Oh, and so instead of saying, well, all our logic's in the view layer, so let's get really good at testing the view layer, which means massive device farms and a bunch of people testing all these things. Let's make that layer as thin as possible. And there's analogies for this in even how we do service design these days and structure the architecture of systems.

Basically, make the boundaries as thin as possible and the interaction with the outside world as thin as possible. And that gives you much more capability to effectively test the majority or much larger portions of your business logic. So the device farm problem is still a problem. People still want to see how something specifically renders on a particular screen or whatever. But by minimizing that, the amount that you have to invest in that gets smaller. This episode is sponsored in part by our friends at Optics because they believe that many of you

are looking to bolster your security posture with CNAP and XDR solutions. They offer both cloud and endpoint security in a single UI and data model. Listeners can get Optics for up to a thousand assets through the end of 2023, that is next year, for $1. But this offer is only available for a limited time on OpticsSecretMenu.com. That's U-P-T-Y-C-S SecretMenu.com.

You mentioned Device Farm, which is an app choice, given that that is the name of an AWS service that has a crap ton of mobile devices that you can log into, and it's one of my top candidates for the, did I make this service up to mess with you competitions? It does lead us to an interesting question. CICD has gotten an increased amount of attention lately from pretty much everyone. And AWS, as is typical for Amazon, tends to lie awake at night worrying that someone somehow is making money that isn't them. So their product strategy distills down to yes. So they wound up releasing a whole bunch of CICD-oriented products

that at launch were, to be polite, terrible. And over time, they've gotten slightly better, but it's still a very confusing ecosystem there. And then we see things like Azure DevOps, who it seems is aimed at a very similar type of problem. And they're also trying to challenge Amazon on the grounds of terrible names of services. But we're now seeing an increased focus from the first party providers themselves around the CI CD space. What does that mean for existing entrenched players who have been

making a specialty out of this for a lot longer than these folks have been playing with it? Yeah. So it's a great question. I think about the approaches very differently, which is probably unsurprising. I mean, speaking of lying awake at night or like spending all day thinking about these things like this is this is what we do. So you've used the term script runner a few times in the conversation. The thing that I see when I see someone like AWS looking at this problem is basically, you know, people are using the way that I think about it as maybe less the money, although it translates pretty quickly. People are using compute to do something. You know, can we get them to do that with us? And, you know, oddly enough, a massive chunk of CircleCI runs on AWS, so it doesn't really matter to them one way or another. But they're effectively looking to drive compute hours and looking to drive, you know, a pathway

onto their platform. And so one thing about that is it doesn't really matter to them, in my perspective, whether people use that particular product or not. And as a result, it gets the product investment that you put in when that's the case. And so it's a sort of a check the box approach. Like, hey, we have CI and we have CD like other people do. Whereas when we look at CI and CD, we've been talking about some of the factors like scaling it effectively and making it really easy for you to understand what's going on. We think about very much the core use case, like what is one of our customers or users doing when they show up? How do we do that in a way that maximizes their flow, minimizes the overhead to them of using our system,

whether it's getting set up and running really quickly, like talk about being in the center of how much of the world is developing software. So we see patterns, we see mistakes that people are making and can use that to inform both how our product works and inform you directly as a user. Hey, I see that you're trying to do this. You know, it would go better if you did this and so i think both from the the honestly the years that we've been doing this and the amount that we've witnessed in terms of um what works well for customers what doesn't what we see going through just from a data perspective you know as we see hundreds of thousands of of builds running that rich perspective is unique to us because as you said we're a player that's been doing this for a really long time and very focused on it and we treat the experience with I guess I'm trying to

figure out a way to say this that doesn't sound as bad as it might but a lot of people have suffered a lot with CI and CD there's a lot that goes into getting CI and CD to work effectively and getting it to work reliably over time as your system is constantly changing. And honestly, there's a lot of frustration. And we come in to work every day thinking about minimizing that frustration so that our customers can go spend their time doing what matters to them. And again, when I think you, you know, you sort of, a lot of these big players present you with a runtime in which you can execute a script of your choosing. It's not, it's not thinking about

the problem in that way. And I don't see them changing their perspective. Honestly, I just don't worry about them. Which is a very fair tack to take. And it's interesting watching companies and as far as how much time and energy they spend worrying about competition versus how much they focus instead on customers. So to turn it around slightly, what makes what you do challenging in some respects, I would imagine, is that a lot of your target market is themselves developers. Developers, in my experience, are challenging customers in that first, they tend to devalue their own time to the point where, oh, that doesn't sound hard. I'll build that overnight. Secondly, once you finally win them over to the idea of paying for something,

it's challenging to get them to have the necessary signing authority. So at best they become champions, but what you do has to start with developers in order to win widespread adoption and technical buy-in. How does that wind up manifesting as approach to what some people call it developer relations, developer advocacy. I refer to those folks as dev relippers because I have problems. But how do you folks view that? Yeah, it's a really insightful view, actually, because we do end up in most of our customers or in the environments of our customers, however you want to describe it, as a result of the enthusiasm of individual developers, development teams, much more so than, you know, there are many products in certainly an enterprise software.

And I don't really think purely an enterprise, but there are many products that can only be purchased by the CIO or the CTO or whatever. Right. And so to your question of developer relations, we spend a lot of time out in the market, you know, talking to individuals, talking at conferences, writing content about how we think about the space

and things that people can do. But we are a very product-driven company, meaning both that's what we think about first and then support it with these other things but second we win on product right like we don't win in the market because you thought um a blog post that we wrote was really cool that might make you aware of us but if you don't love the product i mean developers to your point they they want to use things that they really enjoy using. And when developers use the product and love the product, then they champion it and they get access because they might work on a side project or an open source project,

or maybe they worked in another company that used CircleCI and then they go somewhere else and they say, what are we doing? Life is so much better for you circle ci those sorts of things um but it very much comes from the bottom up it's it's pretty difficult you know to go into an organization and say hey you should push this down to all of your developers there's a lot of uh there's a lot of rejection that comes from developers on mandated tooling And so we have to provide knowledge, we have to provide capabilities in our product that appeal to those other folks. So for example, administrators of our tooling,

or when it gets to the point where someone owns how you use CircleCI versus just being a regular user of the product, we have capabilities to support them around understanding what's happening, around creating shared capabilities that multiple teams can use, those sorts of things. But ultimately, we have to lead with product and we have to get into the sort of hearts and minds of the developers themselves and then grow from there. And everything we do from marketing you know, marketing, developer relations, myself, I spend a lot of time talking to customers or out in the market

is all about propping up or helping people, like helping raise awareness effectively. But there's nothing that we can do if the product doesn't meet the needs of our customers. And that's what it seems like it comes down to a fair bit. It's always weird to consider that at its heart, developer relations is marketing. And the folks I talk to who argue against that, it seems that it comes from a misunderstanding of what marketing actually is. It's not buying ads in airports. It's not doing podcast advertisements, which is a subject near and dear to my heart. It's not about annoying people by showing up at their office with the sales team. It's about understanding what their challenges

and problems are, and then positioning a solution that ideally solves them in a place that, and in a way that is, that they can be receptive to. Instead, people tend to equate marketing to this whole ridiculous statistics driven nonsense that doesn't really resonate with anyone. And I think that that's unfair to everyone involved. That said, I will say that having spent a fair bit of time in this space, I've yet to see anything from CircleCI that has annoyed me to the point where I would have remembered it, which is awesome.

I don't see it in in-flight magazines generally. I don't see it on obnoxious people trying to tackle me as I walk through an expo hall and want to scan my badge. It just seems very well executed and you have some very talented people working for you. To that end, you are largely a distributed company, which is fascinating. Did it start that way? Did it happen that way by a quirk of fate? Yeah, those two things probably come together. So the company from very early days, now I wasn't there, but I think some of our earliest engineers were distributed.

And, you know, the company started out basically entirely as engineers, the team solving problems of other engineers, which is a, it's a fun challenge. And so there were early participants who were distributed, mostly, you know, when you start a company and no one has ever heard of you and no one knows if you're going to be successful, going and recruiting is generally a different game than when you're certainly when you're where we are now. There were some personal relations that just happened to connect with people around the globe who wanted to participate. And so we started out pretty early with some distribution and that led to structuring the org in a way, you know, both from a tooling and process perspective, a lot of that sort of happens organically, but building a culture that really supported that, you know, I personally am based in the Bay Area, so we have a headquarters in San Francisco, but it doesn't really make a difference if I go in versus, you know, just stay

and work from home on any given day because the company operates in such a way that that distribution is completely normal. We accidentally did the same thing. My business partner and I used to live across the street from each other, and we decided to merge a week before he moved out of state to Portland. So, awesome. Great. We have wonderful timing on all of these things. It's fun to build that way from the ground up. The challenge I've always seen is when you start off with having a centralized office and everyone's there except this one person who, no matter how you try to work around it is never as involved. So it feels like the sort of thing you've

absolutely got to be building from day one, or otherwise you're going to have a massive cultural growing pain as you try to get there. Yeah, I think that's, I think that's true. So I've actually been that one person. I, um, at some point in my career prior to, to CircleCI was helping out a company founded by some friends of mine based in Toronto. I grew up in Toronto and I kicked off a project and then the project grew and grew until I was the one person out of maybe 50 or 60 who wasn't in an office in Toronto. And it got to the point where no one remembered who I was. And I was like, cool, I think I'm done. I'm out. I was fine with that. It was always meant to be a temporary thing,

but I really felt that transition for the organization. And I would say in terms of growing, I mean, yes, if you start out, it kind of goes both ways. If you start out distributed, you're going to remain distributed. There are certain things that get more challenging at scale, right? Um, if everybody is sort of just in their home, um, all over the globe, then the communication overhead continues to increase and increase. And just understanding who, um, who people are, who you should be talking to. You need to focus. There's always a time zone hierarchy too. Oh, the time zones are a delight. Yes. And I would say like we talk a lot about in this industry, Dunbar's number and sizes of teams and the points at which things get more complex. And I think there's probably a different scale for distributed teams. It takes fewer people to reach a point

where communication gets challenging and trust and all the other things that go with sort of Dunbar's views, you know, you kind of have that challenge and then you start to think, oh, well, you know, then you have some offices because we actually have maybe six physical offices, partly because in our go-to-market org, you know, we've started to expand globally and put people in regional offices. So there's kind of this interesting disconnect. I don't know about disconnect, but there's a split, right, in how we operate in different parts of the org. And I think what I've seen people, well, I don't know about succeed, but I've sort of seen people try when you start out with one org or sorry, one location is let's not jump to like that one person somewhere else and then one person somewhere else

kind of thing, but build out a second office, build out another office, like pick another location where you think you, it's often certainly where we are in the Bay area. Um, it's often driven by just this market, right? Finding, finding talent, finding, uh, people who want to join you, hanging on to those people when there are so many other opportunities around tends to be much more challenging. And when you offer people alternatives, like you can stay where you are, but have access to a cool and interesting company, or you can work from home, which a lot of people value, then there's different things that you bring to the table. So I see a lot of people trying to expand in that way. But when you are so office centric, a second office, I think, is a smoother transition point than just suddenly distributing people, because especially the first and second one, unless you're hiring in a massive wave, are really going to struggle in that environment. I think that's probably one of the more astute things that's been noticed on this show in the last couple of years.

If people want to hear more about what you have to say and how you think about the world, where can they find you? I would say on our blog, I tend to write stuff there as do other people. There's a lot of, you talked about having great people in the organization. We have a lot of great people talking about how we think about engineering, how we think about both engineering teams and culture, and then some of the problems we're trying to solve. So off our site, circleci.com, go to our blog. And then I tend to speak and hang out on podcasts and do guest writing. I think I'm pretty easy to find. You can find me on Twitter.

My handle is ZubZ00B. I'm not super prolific, but if someone was to track me down and ask me something, I'd probably be more than happy to answer. And you can expect some engagement as soon as this goes out. Thank you so much for taking the time to speak with me today. I appreciate it. Yeah, thanks for having me. This was a ton of fun. Rob Zuber, CTO at CircleCI. I'm Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. If you've hated this podcast, please leave a five-star review on Apple Podcasts, along

with something amusing for me to read later while I'm crying. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS, we tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started. This has been a humble pod production stay humble