ACM ByteCast - Theo Schlossnagle - Episode 5

Episode Date: September 15, 2020

In this episode of ACM ByteCast, Jessica Bell talks with widely respected industry thought leader Theo Schlossnagle, Founder and CTO at Circonus, Co-Chair of ACM Queue, member of the ACM Publications Board, and elected Member at Large on the ACM Council. He shares his journey from a love of problem solving to computing and entrepreneurship and how his work at school helping classmates solve difficult problems led him to create his first company and provide solutions to some of the biggest internet businesses in the world. He also provides valuable advice for companies just beyond the start-up phase and to young engineers interested in founding a business.

This is ACM ByteCast, a podcast series from the Association for Computing Machinery, the world's largest educational and scientific computing society. We talk to researchers, practitioners, and innovators who are all at the intersection of computing research and practice. They share their experiences, the lessons they've learned, and their own visions for the future of computing. I'm your host, Jessica Bell. All right. Today, I would like to welcome Theo Schlossnagel, who is a serial entrepreneur,

longtime ACM volunteer, and gentleman farmer. He also just happens to be the founder and CTO at Zirconis. Theo, welcome to ACM ByteCast. Thank you for having me. Well, first off, for people who aren't already aware of you, can you give us our intro to Theo pitch? Pitching myself. That's something I'm very good at. So I've been doing computing for quite some time. Started my interest in that, I guess, when I was in middle school, but ended up going to Johns Hopkins for computer science, stayed there for far too long, left with far too little, and started a couple of different companies, beginning with a professional services and consulting company around internet scale growth in 1997 called OmniTI. Awesome. And I'm always curious about why people get into computing.

Were you the sort of typical story of, oh, I was taking apart computers when I was young, or did something particularly draw you to that field? Or yeah, what brought you to tech? I don't know. I mean, I guess relatively young. I would say when I was about eight or nine, my parents got me an Apple IIe computer. And when I found out that I could

program it in BASIC, I just found it to be really interesting. I draw a lot of parallels between computer programming, which indeed is different than computer science, but a lot of parallels between computer programming and puzzle solving. And they use the same portion of the brain. And I find it to be stress relieving to be frustrated in that way. I like doing puzzles. I like writing code. I think they're sort of the same thing. Interesting. That's such a on point view of code. I remember when I was first learning, I had this little puzzle game where you would snap in the different statements together to sort of think about logic and programming. And yeah, that's an interesting way of putting it.

I like that. understand enough of how computing works to really understand what's going on when you, you know, write a program and interpret it or compile it to machine code and operate it and run it. After you take in like a computer systems architecture class and you kind of know how the chips light up. I started as an electrical engineer. So a lot of gate design, BSLI design. So I always find it most fascinating when things don't work the way that I intuitively expect them to. And I usually get called in to look at kernel bugs and compiler bugs and regular bugs and things like that.

So I find that violating the layers of abstraction that are required to make computers useful and productive tools for people in order to solve problems to be just an absolutely fascinating journey. So is that journey what ended up leading you into working with architectures and sort of scalability? Talk to us a little bit about that journey from what could have been an academic career path, but sort of went into founding an entrepreneurship path. I think it was actually an outcome of opportunity more than anything else. So I ended up graduating from Hopkins, I think in 97, and then with a master's in 99, and then I pursued a PhD until 2003, but left without it. And during that timeframe, as everyone knows, that was when things got pretty exciting on the internet.

So it was an incredible shortage of expertise in the realm of large-scale distributed systems and high-performance code. And I had some of that expertise through my research at Hopkins. I was working on distributed systems things, specifically on load balancing algorithms and techniques. So it was directly applicable to the large-scale web. And as all of my peers in school graduated and went on into industry, when they ran into problems and they were short-staffed, they kind of lobbed my name around often as someone to call in as backup. And I ended up

founding a company around that and was pulled into some really exciting engagements with some of the biggest internet companies in the world. So I got to see a lot of really interesting problems. And again, it's like looking at a puzzle. Things don't work the way you expect and you need the pieces to fit together differently. It's a great mental exercise that I find really satisfying. So talk to us a little bit then about your experiences between the beginning of that career when you're being called in to solve these sort of issues that might be down a couple layers of extraction. Have you seen a major change of what kind of puzzles are happening 10 years ago, 15 years ago versus now? As the internet grows, as we see a lot more of these very large scale companies,

has it changed or is it essentially the same problems? I think the problems are very, very similar. The problems in distributed systems are more prevalent now because most of the systems that we use, if you deploy into the cloud, there's basically no way that you can't have even a traditional distributed system. But even if you're writing an iPhone app,

your iPhone is a distributed system. It's got all these different processors on it that run at different speeds and different buses and all sorts of stuff. I think that one of the things that gave me a distinct advantage of being able to apply my skills to industry is that when I started at Hopkins, I ended up working in the computer science lab as a technician, sort of, or a systems administrator, I would make sure the systems ran, got installed and upgraded and patched, make sure the students weren't exercising exploits, which they were always trying to do, that sort of thing. So a lot of grungy operations work. So when I went to apply computer science principles to problems, it wasn't just an idea. There was the ability to go in and recompile all the apps and change the code and deploy it and understand how those

systems, not only were they designed or written, but how they were deployed and operated as well. It sort of allowed me to take my ambitions to the finish line. I think that one of the challenges that I think is fantastically eroding, but one of the original challenges is a lot of people who wrote code didn't know how to provision servers and rack them up and get the networking working. And they needed a team, an IT team to be able to enable the deployment and operationalization of their code. Today with the cloud, this whole kind of no-ops movement where you can sign up for an Amazon or Google account or a Microsoft Azure account

and kind of deploy your code with a couple of clicks has removed those barriers. But I think that it's kind of short-sighted to think that that's like how Netflix runs, right? I mean, Netflix and Google and Microsoft and Amazon, they all hire data center operations people. They hire hardware specialists. A lot of them make their own chips. They also really understand how the underlying components that drive the services that they use work.

And they build them. So like kernel engineers and things build them. So like kernel engineers and things like that. So the world's a much more complicated place. And I think that the biggest challenge that people run into today is that when you struggle to solve a problem, when you hit a bug, when you hit unexplained behavior, the layers of abstraction have become more difficult to penetrate. So if I have a kernel bug and I'm running in the cloud, like I can't see that kernel, you know, it's very, very difficult to get access to the kernel that's running in the VM or hypervisor bug or things like that. So those are different challenges. Do you feel like the trade-offs in accessibility on writing

your code and getting it deployed via these one-click or cloud platforms, do you think those trade-offs are worth it? Or is it kind of a it depends on your situation question? Absolutely situational. But I would say that 99% or more of the time, it's totally worth it. If you can increase developer productivity and take new ideas to an audience, like actually take an idea to an audience for use with less friction, then that's a huge win. And then again, if you start running into problems where those abstractions are a hindrance, the only reason you'd really run into those where it matters is if you have an audience, you have demand for your services or your application

or your code. And then, you know, that's kind of a problem you want to have. It's great to tackle it at that time. Yeah. What kind of things would you say to a company that is beyond the startup stage? So not, you know, the scrappy little, we're trying to get anything done by the skin of our teeth, but like, okay, we're ready to scale up a little bit. What are some of the common mistakes that you see or the common questions that people fail to ask themselves at that stage? So most code and infrastructure and design patterns for architectures of code resemble organizational structure. So you should take a really hard look about what your organizational structure looks like, how you think it's going to look in a few years after it grows a little bit more, what you don't like about that, and realize

that all of that stuff that you don't like will also manifest itself in your code and your architectural designs. I think that there are a lot of trends around, you know, monoliths versus microservices. Using the best tool for the job is a great example of that as well, where people always want to use the best tool for the job. But if you have 40 employees who all decide that a different tool is the best tool, then you have no shared domain knowledge over those tools. And you have these severe friction, impedance mismatches when handing code or doing code reviews. So it's sort of an organizational dysfunction, even though it's a local optimization. So a lot of this, again, local optimizations and organizations usually have pretty

bad consequences on a macro level. And the exact same thing is true for code. So I would say that discipline and structure and a focus on making sure that your organization skills are solid, you know where your things are, you know where your code is, whether or not you use test-driven design, for example, which I personally am not a fan of. And every time somebody argues with me, they go and redefine test-driven design in a way that's not objectionable. But like the concept of like, I'm going to write my tests before I write my code, I'm not a huge fan of. I absolutely think that code must have tests, but you have to know where your code is, where your tests are, what types of coverage you have, where your confidence lies, all of that stuff are

really organizational skills. And strong organizational skills that have kind of uniform adoption across an organization will prevent you from getting into places that you can no longer get out of. Where do you think some of these mismatches come from? Is it a fact of our industry is kind of like startup and we'll just do everything as fast as we possibly can without a little bit of intention? Or does it end up being a fracture in the kind of developers or programmers that you're hiring?

Talk to me where you see those cracks kind of coming from. I would say that in immature and pseudo maturemature organizations, so not talking about like Amazons and Googles and things like that, though occasionally they manifest these problems, the concept of the inmates running the asylum is incredibly apt. There are a lot of technical challenges where people feel that they need to hire exceptional programmers for, and who are they to say how they should do their job? And if you can imagine having a complicated finance situation where you hired 30 finance people and said,

don't tell me how to do the books, right? You each individually decide what the best way to do the books is, right? Enjoy your audit, right? I think that's going to be a bad situation. Yet in computing, you know, all of the engineers, definitely not all of the engineers, but a lot of engineers that I've interacted with that manifest these problems feel that they were hired because they know best and they should be entitled to make all of these decisions on their own. And that never makes sense. And especially in a large organization, that lacks any sort of uniformity, which is very, very dangerous. So I think that at the end of the day, it really comes down to simply the inmates being allowed to run the asylum. And that, again, you want input and

you want new ideas. And this industry changes ridiculously fast. So you need that infusion, but you need control over that. You need like a gated introduction of that stuff. It needs to be carefully deliberated before it's brought into an organization. You don't want 14 different language frameworks in an organization with 20 different people. That's an enormous problem. And it's arguable that you may not want 14 in an organization with 500 people. I'd like to hear a little bit more about your founder experience as well. Do you feel like that is something that just happened out of necessity to find clients to match your skill set? Or is it something that you were drawn to as part of a entrepreneurial spirit? I think it's more of the latter. Well, when I first started, there weren't really any of the big tech companies in the way that they look now. So I mean, if I was starting my career now, I think that having a job at, you know, an exciting tech centered company that isn't a tech company, right?

So you've got high frequency trading companies, you've got finance companies like Bank of America, you know, things like that, that are not tech companies, but they are incredibly tech centered now, right? So that old saying that every company is a tech company was absolutely not true in 97 when I started my first company. And what I found was that I was really excited about doing technical things, applying computer science concepts, making sure that things were maintainable and operable and things like that. But I didn't want to do it for the sake of computing. I wanted to do it for the sake of other businesses. And I found that I was being grossly underpaid to do

that at the first company that I worked at. So I flipped and started my own, mainly because I thought I could run a better company that I enjoyed working at more. Yeah, I feel like I was right. So do you feel successful in that? Yeah, I founded four companies, one of which is closed. Well, I guess I actually shut down the first one last year as well. But they were all relatively successful. Three of them probably combined had 500 or 600 employees across them. and for employees. A lot of livelihoods created and got a lot of feedback from, you know, not everybody, but many of the employees that they found that the experience of working in those companies was foundational in their careers. So I am very satisfied with the outcomes of that. Yeah. Do you think your sort of programming thought process helped or hindered in the entrepreneurial set of starting a company,

being a founder? And then do you think vice versa? Does your founding experience help or hinder your sort of programming brain, if that makes sense? I think that there's a little bit of overlap. Quite honestly, I think that it's very rare to find one person who is all things and a sales and growth marketing person, I am not. So while you can grow a company in the beginning on evangelism, most companies, in order to hit a second level of growth, really need to have somebody who is focused on that, who's focused on growth, focused on marketing, focused on sales, sales process, training, onboarding, all of those things. These are not technical things. They have nothing to do with computer science. They're very business focused and there's only so much room in your

brain. So I would say I'm not very good at those. And being a computer scientist hindered the ability to grow only in that I had to hire someone else. I think it was very important to that self-recognition that, Hey, look, I'm not good at all of these things. I had to hire someone else. I think it was very important to that self-recognition that, hey, look, I'm not good at all of these things. I need to hire people who are better than me. My fantastic vision of every company is to figure out anything that I'm good at and find somebody better and hire them. I don't want to be the best at anything in my companies, right?

It just means I didn't hire the right people. But I would say that from a founder's perspective on how it positively influenced my engineering, I would say that from a founder's perspective on how it positively influenced my engineering, I would say that the customer is always right. While that's not a perfect statement, it is incredibly important centering thesis. And that like the test-driven design sort of thing, it's one of the reasons why I find code testing to be a little misguided in a lot of ways. It doesn't need to be, but it tends to be in

practice is that you can write a hundred tests to make sure your code works. But the only real test is, do I have a satisfied customer that wants to return? And it's very hard to write a test to test that. But that is ultimately what needs to be tested when you do a commit and keeping that perspective throughout the entire process of building new products and maintaining old ones has been incredibly valuable. Right. Technology at its core is a people issue, not a machine issue, I feel like. Yeah, it doesn't matter if it's fast. It only matters if it serves its purpose and is useful to people.

Right. What would you say to the group of engineers who are coming out of their technology programs, coming out of jobs at big technology companies and want to go down this same founder path? Do you have any tidbits of wisdom or encouragement or maybe not encouraging at all? I won't say that people shouldn't do what I did. That would be hypocritical. I would say that there are a lot of aspects of founding at a company that are very non-technical and very difficult. So there's a lot of stress in the financial aspects of it. So the first two, three companies, first three companies that I founded had no venture capital funding. So, you know, there were liens against my mortgages. If I screwed up, I lost everything. And when you

hit hard times and people don't tell you hard times, but hard times are not always when you can't make a sale. It's when you've made a sale, you've done the work and you can't get paid. Right. These are like really not technical issues at all. These are business issues. And then you don't have enough money to pay people. So you have to pay your employees before you pay yourself, right? So like all of these things are stresses that are difficult. So you need to be able to cope with stress. Well, you also can't really talk to your peers in your organization about your challenges. It's the old saying, it's very lonely at the top. It's very true, right? So you need a network of support that's outside of your company, either through fellow founders, fellow CEOs, or just a family

and peers that are outside of your direct report chain or industry. So there are a lot of challenges that are around entrepreneurialism that are very, very real and they're very long-term. So it's a hard process, but I would say that it's an incredibly rewarding thing when I look back and people ask what I'm most proud of. So I ended up founding a company called OmniTI first, and then a company called Message Systems, which is now called SparkPost, and Zirconis, and then another one called FontDeck, which is no longer around. And I think SparkPost today, I think delivers probably 30 to 40% of the world's email traffic. So if you use LinkedIn or Twitter or Amex or your Chase

Visa card or whatever, the emails that you get go through that software platform. So you might think that I'm very, very proud of that. I'm actually really proud of the positive feedback I've gotten from so many employees over the year and the number of livelihoods I've helped create. So that to me is something that I could not have done if I was a line level employer or even a manager. Yeah, don't think about that part. Great. Well, in our last couple of minutes, I always like to hear from people a little bit about the future, what you're worried about, what you're thinking about, what you're excited about. I find that very curious for people who have been especially for people who have been in the industry a lot longer than I have. I'm a

relatively new technologist. So yeah, what do you think about technology at night? What are you super excited about? What are you kind of scared about? Tell us about the future of Vio. Yeah, I mean, there's so many things. We're really in the midst of an enormous transformation. The fact that I can talk to most of the devices in my house and they actually understand me even when I mumble, that's truly amazing. If you rewind 15 years, that was unconscionable. That was AI then. I mean, we realize that's not AI now. So that opens technology and productivity and convenience levels to the world in a way that hasn't been there before. Things that concern me, there's always this constant backdrop of the ethics of computing. So the competing has great power to

disenfranchise people just as it has power to enable people. And if we're not very careful as engineers and product managers and companies building the software, we are giving tools to kind of manifest the destruction of sections of society, if not society itself. And that sounds very doom and gloomy. And the only reason that I don't think it's doom and gloom is that we really don't want that, but the potential is there. So we really need to be careful and have kind of a consistent adoption and focus on ethical practices in software engineering and software product design as well. That's kind of a constant backdrop. On the more technical level, I would say that computing is producing machine data in a way that is just ginormous. I mean, like there's just no word for the size of the machine data that's coming

out of systems now. So I expect in 10 years, your smallest IoT device will be able to produce a gigabit a second of questionably valuable machine data. And the question is, what do we do with that? Where do we put that? How do we change that into value? That's one of the things we focus on at Zirconis, the amount of data volume that we've seen when we started the company. We started the idea of the company probably 15 years ago. We have one times 10 to the maybe 12th or 14th more data coming in per second now than we did then. Oh my gosh.

So we have trillions of samples per second coming in from devices around the world. And you have to make sense of that and produce value out of that and do it economically because that device, sometimes they cost 25 cents, right? And they're producing data that in the old days would cost tens of dollars a month to process. And like, there's no economic model around that. So I think that it's a constant leapfrog of like technological advancements, business use case advancements, economic advancements. And then there's this thing that I feel like it's been ignored, which is this ethical advancement of should we really do this? What are the consequences of doing this? Right, right. Exactly.

Great. Well, I don't think that was all doom and gloom. A little bit of both. Excellent. Really nice to talk to you. And thank you for all of your insights and recounting on your history. And thank you so much for joining us. Well, I really enjoyed it. Thank you.

ACM ByteCast is a production of the Association for Computing Machinery's Practitioners Board. To learn more about ACM and its activities, visit acm.org. For more information about this and other episodes, please visit our website at learning.acm.org slash b-y-t-e-c-a-s-t. That's learning.acm.org.