The Pragmatic Engineer - Python, Go, Rust, TypeScript and AI with Armin Ronacher

Episode Date: October 8, 2025

Brought to You By:•⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more. Most teams end up in this situation: ship a feature to 10% of users, wait a week, check t...hree different tools, try to correlate the data, and you’re still unsure if it worked. The problem is that each tool has its own user identification and segmentation logic. Statsig solved this problem by building everything within a unified platform. Check out Statsig.•⁠ Linear – The system for modern product development. In the episode, Armin talks about how he uses an army of “AI interns” at his startup. With Linear, you can easily do the same: Linear’s Cursor integration lets you add Cursor as an agent to your workspace. This agent then works alongside you and your team to make code changes or answer questions. You’ve got to try it out: give Linear a spin and see how it integrates with Cursor.—Armin Ronacher is the creator of the Flask framework for Python, was one of the first engineers hired at Sentry, and now the co-founder of a new startup. He has spent his career thinking deeply about how tools shape the way we build software.In this episode of The Pragmatic Engineer Podcast, he joins me to talk about how programming languages compare, why Rust may not be ideal for early-stage startups, and how AI tools are transforming the way engineers work. Armin shares his view on what continues to make certain languages worth learning, and how agentic coding is driving people to work more, sometimes to their own detriment. We also discuss: • Why the Python 2 to 3 migration was more challenging than expected• How Python, Go, Rust, and TypeScript stack up for different kinds of work • How AI tools are changing the need for unified codebases• What Armin learned about error handling from his time at Sentry• And much more Jump to interesting parts:• (06:53) How Python, Go, and Rust stack up and when to use each one• (30:08) Why Armin has changed his mind about AI tools• (50:32) How important are language choices from an error-handling perspective?—Timestamps(00:00) Intro(01:34) Why the Python 2 to 3 migration created so many challenges(06:53) How Python, Go, and Rust stack up and when to use each one(08:35) The friction points that make Rust a bad fit for startups(12:28) How Armin thinks about choosing a language for building a startup(22:33) How AI is impacting the need for unified code bases(24:19) The use cases where AI coding tools excel (30:08) Why Armin has changed his mind about AI tools(38:04) Why different programming languages still matter but may not in an AI-driven future(42:13) Why agentic coding is driving people to work more and why that’s not always good(47:41) Armin’s error-handling takeaways from working at Sentry (50:32) How important is language choice from an error-handling perspective(56:02) Why the current SDLC still doesn’t prioritize error handling (1:04:18) The challenges language designers face (1:05:40) What Armin learned from working in startups and who thrives in that environment(1:11:39) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Transcript
Discussion (0)
Starting point is 00:00:00 How do you think about Python ecosystem today, the Rust ecosystem, and the Go ecosystem? The Python ecosystem is a lot of infrastructure, a lot of provisioning machines. Rust, I think, if you work with binary data, if you build a load balancer, you build a database. Go, in particular, I think it's just a good language for building web services and really kind of only web services. Speaking about AI, agentic coding, how are you using them? I had to cloud build me my perfect control system to get my logs and, visualize what's going on in production. And I would never have done this before just because I wouldn't have work. Why have you become so much more positive about these AI coding tools? So the biggest thing is
Starting point is 00:00:38 that Armin and Ronautier is a widely known open source contributor and the creator of Flask, a popular Python-Wed framework. He was also engineer number one at Century and is now building his own startup, making heavy use of AI tools. Today with Armand we cover why AI coding tools are making the choice of programming languages more important and not less. Python versus Go versus Rust versus TypeScript and which languages to use for startups and why. What Armin learned about error handling after 10 years a century and why typesafe languages like TypeScript don't seem to reduce errors, and many more. If you're interested in understanding more about the strengths and weaknesses of programming languages, how LLMs are changing how startups are built or want to know more about error handling,
Starting point is 00:01:17 this episode is for you. This podcast episode is presented by Statsig, the Unified Platform for Flags, Analytics, Experiments, and more. Check out the show notes to learn more about them and our other season sponsor. Let's jump in. Armin, welcome to the podcast. Hi, happy to be here. So let's talk about programming languages. You've been very deep into Python for many, many years, well over a decade, and now you've touched on other languages.
Starting point is 00:01:43 But with Python, how have you seen the Python itself change? And can you give us, for those of us who are not as in-depth in Python, give us a bit more detail about the two-to-three migration drama, which I think if you work with anyone who work with Python, you've heard them moaning. I was at Uber when this happened, and there was a lot of back and forth, a lot of delaying.
Starting point is 00:02:09 It seemed very rare for, across any languages, to see what happened from Python 2 to Python 3, which seemed like breaking changes, lots of disagreements, some people just wanting, a lot of very competent and good engineers wanting to stay on the kind of two, which is lower than three. So what happened there?
Starting point is 00:02:30 Yeah, so fun fact, this year was the Python documentary, which Guido invited me to be part of, which was very nice. And I had that documentary had a segment on it, on that Python 2-2-Certains three migration, and that basically made me go back in time just to remember all of it. Because in many ways, like my memory was also incredibly fuzzy and hazy. I also remember certain parts more than others.
Starting point is 00:02:59 So if you really want to go into the deeps of Python's 3, look at the Python documentary, it has a whole segment on it. But I think like in retrospective, I think you can't do it anymore. It was just at this moment in time, it didn't kill Python. But I think it could have, right? If people didn't put a lot of energy in to actually make that migration work, it could have been quite problematic for the language. because if you go back to Python 3-0 and 3-1 and 3-2,
Starting point is 00:03:30 there were so many missteps in the original library designs and decision to be made. And so as someone who maybe hasn't participated in this originally, you have to consider that Python started out with just basic byte strings like C. Eventually, it gained unical strings as an option. So you give it a U in front of the string and it became a unical string.
Starting point is 00:03:54 And Python's biggest motivation of existence was to be a lot more strict about strings and move everything to Unicode. What wasn't quite anticipated is that that migration was a little bit more nuanced in practice than designs, right? There was a very simplistic view of what Unicode looks in the real world. And it didn't look like that. and I actually think that Python's messy Python's remigration also had a really positive impact on
Starting point is 00:04:27 other ecosystems like Ubuntu for instance because there were like two things sort of happening simultaneously Python's 3 got a little bit more restrict, a bit more lenient to working with non-unicode data
Starting point is 00:04:41 but also a bunch of the situations where Python made it very hard to work with Unicode was actually all kinds of configuration issues. So like I remember not every Linux had UTFAid everywhere. It was still very common for you to connect to university network and not find a UTIF8 Linux, which had all kinds of really funny things on a file system, right?
Starting point is 00:05:04 So if these things converged a little bit, so the Unicode story got better. But you had millions of lines of Python 2 code that couldn't work on Python 3, and the initial assumption was you can just migrate once and then you're Python 3 and you're good. And that's not what happened. We had to maintain libraries simultaneously
Starting point is 00:05:23 for Python 2 and Python 3 for many years. And I, for instance, I remember meeting at the Python language summit in Florence, where I proposed to bring back the U prefix on unicode strings so you can have them optionally. Because I noticed that if they were there,
Starting point is 00:05:41 I could write code that supports both 5 and 2 and Python 3. And I still remember there was a huge pushback. It was like, why would you even want to maintain a library for both two and three. You should just move to Python 3 and then make everybody move like that. But that just doesn't work like that. So what really made the Python 3 migration
Starting point is 00:05:58 ultimately work was a lot of effort put in by a lot of people and just a more realistic look into that you have to, it took 10 years, right, or more even. You have to just support both versions in one form another. That is what made it work.
Starting point is 00:06:18 but I think the Python's remigration also left a pretty important data point for future languages to approach this better. And I remember Rust early on specifically pointed at the Python's remigration to demonstrate why it wants something like an addition system, or I forgot what was called originally, but basically a very targeted opting into new features or opting out of all features so that you can have code based. even within the same project from different versions. Nice. And speaking of the differences in programming languages, you mentioned Python has been, was your favorite language for a very, very long time. Later, you helped introduce Rust into Sentry, partially for performance reasons as well. And now we're also playing with Go.
Starting point is 00:07:06 How would you compare these languages? And like, when you think of a language, what is your mental model of like, what is this language good for, not good for? one would I use it for? I wrote a bunch of blog post that you're not quite about languages, but they're sort of about my general split brain in a way, because I have two programmers in myself.
Starting point is 00:07:28 One is I like building cool open source software that hopefully thousands of people use and put a lot of handcrafted efforts into it. It's like the Swiss watchmaking of source code writing. And there is like, I care a lot about the language, I care about a lot of the API and everything. But then when you build a company, when you build a product, none of that matters.
Starting point is 00:07:47 is a write once run many times code, but not a right once, write many times against the code code, right? And so what makes Rust amazing, I think, for crafting really cool open source code also makes it a suboptimal programming language for a startup because there's much more friction in it, right? It's a much more precise language.
Starting point is 00:08:14 It's a great lift-up compared to C++, which is, in many ways, what the alternative would have looked at at Century for binary file processing. But it's a much less capable language for a startup when it comes to rapid iteration. Can you give examples of this friction? Compile times.
Starting point is 00:08:34 Like Rust compiles so incredibly slow. Speaking about tool choices at startups and scale ups, here's a friction point that hits most growing teams. You're running feature flags in one tool, analytics in another, and trying to debug user behavior. in a third. When you want to know if a feature actually improved conversion or if something that you shipped is causing a bug, you need to manually assist together data from these different
Starting point is 00:08:56 dashboards and then hope that the user segmentation matches up. Most things end up in this situation. You shift feature to 10% of users, then wait a week, check three different tools, try to correlate the data, and you're still unsure if it worked. The problem is that each tool has its own user identification and segmentation logic. That's where Statsick comes in. Satik is our presenting partner for the season, and they've solved the tool integration problem by building everything with a unified platform. Instead of stitching to great point solutions, you get feature flags, analytics, and session replay all using the same user assignments and event tracking. So you ship a feature to 10% of users and the other 90% automatically become your control group with the same event taxonomy. You can immediately see conversion rate differences between groups, drill down to see where treatment users drop off in your funnel,
Starting point is 00:09:43 then watch session recording of specific users who didn't convert, to understand. and what went wrong. The alternative is running ETL jobs to sync user segments between your feature flag service and your analogous warehouse, hoping the timestamps align, and then manually linking up the data that might have different user identification logic. That's a lot of work. Statsic has a generous free tier to get started, and pro-pricing for teams starts at $150 per month. To learn more and get a 30-day enterprise trial, go to Statsic.com slash pragmatic.
Starting point is 00:10:13 There's a huge factor. you write a lot more code compared to Python. You spend a lot of time thinking about types. I love types. I think they're great. But certain things are incredibly hard to express with types. And I don't even have to think about it in Python. Right.
Starting point is 00:10:28 Working with dynamic. And I think one thing to consider here is like dynamic and static languages have aligned a little bit more over the years. I remember the dynamic keyword learning in C-sharp was a great thing because it showed people that can have a statically compiled language with like opt-in dynamic runtime typing for instance
Starting point is 00:10:47 and TypeScript took that entire idea and sort of applied it to JavaScript so you can have both worlds in a way but Rust is like very straightforward like there's just static types and if your problem is a dynamic type problem then you're basically going to create your own dynamic type wrapper left and right it is also that the Borechecker
Starting point is 00:11:08 as great as it is is a huge problem because it doesn't allow to express certain things that other programmers expect there to be, like a chastened boros. So you cannot have a structure that boris into itself. Coming from C++, there's like, why the hell? Like,
Starting point is 00:11:26 why does it do that? Here's my problem. I want to solve it. And the compiler says no. It's like, no, you're wrong. It should work. So that is like, you can run yourself. You can program itself in the corner where it's like incredibly hard to come out. So I think from that perspective, there are a lot of problems which are not rust-shaped. There are a lot of problems which are, in fact, rust-shaped, but a lot of them
Starting point is 00:11:48 are not. And so one of the reasons I'm using Gove right now for the new company I'm working on is because it is a very pragmatic language. And if pragmatism is what you're after, then why not? And you can expect it to hang around and it's not super sexy. And it, like, worst case, even if Google stopped maintaining it, you can imagine there will be dozens of other people who want to keep it alive. Even more than Java is actually an awesome language. It has virtual frets. It makes me happy. Like, I don't have to deal with promises all the time. So I have, I think, a much more pragmatic view when it comes to what a company should build its stuff with because the focus is not so much the code. It is the product that you're building.
Starting point is 00:12:27 So you're now doing a startup. Is it a few people? Is it yourself? So at the moment, it's me and a co-founder. But we just started hiring. And I also like, it's me and a co-founder and a bunch of clothes. and a bunch of codexes and everything. So the world has definitely changed in a sense that there's an army of interns that also now writes code. And that has also changed the way I'm looking at things. So now it's you and your co-founder with a startup. You've got an army of clods and codexes.
Starting point is 00:12:58 And you're expecting to grow the team to hire. As you've settled on the technology you said in this case, this is a lot of it will be go. But what does the programming language need to give you? And for example, like, why was, Python is very flexible. Okay, it's a bit slow. But as you said, it was a good choice. What made you, what would you say like, yeah, go is this good middle ground between Python and Russ and so on? Yeah.
Starting point is 00:13:28 So right now, I think if you're built any company at all, you're going to end up with Python in there if you want it or not. It will be impossible for you build a company. It doesn't have Python code in it. Do you want to build your main service with it? Probably not. But if you do anything with ML, if you do anything with data processing, PYP is going to be in there.
Starting point is 00:13:50 And you can write nice code in Python 2, right? So it's like, that's going to be there. Good luck not having JavaScript in the company. And if you have JavaScript, you have TypeScript. So those languages are going to be there. Another question is like how much of that code that you write overall is in those languages? And I actually felt that I don't want to build backhand services
Starting point is 00:14:09 in either one of those languages. And not even for performance reasons, but just for ecosystem, like what the ecosystems are good at. In my particular case right now, a big part of what I'm doing at the company is parsing emails. And that's just not something JavaScript is amazing at. And something actually Python is quite good at, but I think at scale is Python a good choice here?
Starting point is 00:14:34 And I looked at it and I made a decision that I wouldn't. And from the experience of sort of doing Rust in that space at Century, I felt that I can totally imagine it at scale. Like Rust will make its way back in. But at the moment, I don't think that that's the right tradeoff. And when you say at scale, this is just thinking about when you will have to process huge amounts of data, lots of process running and so on. We're talking specifically about the performance capabilities of the language. I think it's at scale of a company. and that is not necessarily measured by how much data goes through it.
Starting point is 00:15:11 If you scale a company either by the number of people working on it or the complexity of the problem or the complexity of how much data goes through it, you have to make trade-offs, and sometimes that means introducing a new language. Either because of performance reasons or because you integrate into some ecosystem that's particularly strong in some certain environment, and then all of a sudden the language is here. Why did Rust exist a century?
Starting point is 00:15:34 Well, it didn't just exist because I wanted to write Rust. it was like, well, now we have a problem with performance, how are we going to solve it? And on the one hand, it was like, well, we could probably write a Go service here, but it didn't want to maintain another service. So Rust was a pragmatic choice of embedding it in Python. And then later on, when we did native symbol processing, the alternative to Rust basically would have been C++. And for my experience of working with C++, we used C++ on that project first, we just did a lot of crashes. I just didn't feel like I want to maintain that. And so then there was no choice of using Go, because there was no Go ecosystem.
Starting point is 00:16:14 So unless I would have built everything from zero, it wouldn't have worked. But Rust was growing, and Rust had the same problem in the compiler. So the Rust ecosystem for debug files was really good. And there was another company also working on it, which was Mozilla at the time. So there was already two shared interests in this. And that was a really pragmatic choice, even though the language was Rust. Then maybe we over did it a little bit by putting Rust into play. places where it wasn't the most optimal choice, like the ingestion system.
Starting point is 00:16:41 I think at this point probably wouldn't make that decision again. But it would have been non-pragmatic at that moment to say, like, you know what they're going to do? We're going to go into a corner for like nine months and we're going to build this all in this now. Because for the size of the team at the time, the problem was too big, right? And so we had to reuse something. So that's one of the reasons I think why I can imagine that my company is going to, again, end up with adopting a language further down the line that it kind of initially said no to,
Starting point is 00:17:06 just because the situation makes it necessary. And it doesn't necessarily have to mean that performance is the reason. It cannot just be ecosystem stability or anything like that. And speaking about ecosystem stability or ecosystems, how would you think about Python ecosystem today, the Rust ecosystem and the Go ecosystem?
Starting point is 00:17:23 Python ecosystem is a lot of infrastructure, a lot of like provisioning machines. So that's why I'm using it for right now. using preliminary with Python to get the infrastructure up. It is really well entrenched in machine learning. So I have a little data pipeline going on that uses Python. It is also pretty good still at bringing services, web services, to some of the things Python's good at.
Starting point is 00:17:52 Would I necessarily write than the main application logic in it? Maybe, right? Because if I'm doing a whole AI first company and all it does is inference, I'm just waiting on a network layer anyway. So maybe putting some things together with Python is great. Particularly also if you start to do authentic things where you want to have code execution on demand, maybe that's a good choice.
Starting point is 00:18:14 So I think Python has a great future in it. I just don't think it's the most natural language to pick for a service where you think like it's going to be a lot of that. Like higher throughput. I think at this point, Go is an easier language to write than Python for a bunch of engineers,
Starting point is 00:18:34 because Python's increased in complexity and Go didn't quite as much. So I find it to just be a better trade-off for complexity, too. Rust, I think, like, if you have, if you work with binary data, if you maybe, if you have a, you build a load balance, or you build a database. Well, maybe database is now people talk about Sikh and stuff. But, like, if you build like a, like a rust-shaped problem, which I think is largely defined as you single-binary,
Starting point is 00:19:04 to distribute for some reason or another. It can't deal with the garbage collector. You want very predictable performance. You really care what data layout, although at the sake also good. The rest of the benefit that if you also then work in something that requires concurrency and where you're afraid of mismanaging memory
Starting point is 00:19:24 and crashes like modules that they might load into some other high security environment, like a browser, seems like a pretty good choice. working with again like dwarf files like working with just data in general I love Rust
Starting point is 00:19:41 and I think it's is such a pragmatic language also for if you extend Python so if you already have a Python problem and now you have a performance problem or you have an ecosystem integration problem and Rust happens to fit good into that writing extension modules in Python for Python in Rust
Starting point is 00:19:57 beats everything else out there right it's a really good choice for that go in particular I think it's just a good language for building web services and really kind of only web services, maybe some command line tools. One of the other reasons why Rust I think is impossible to get rid of is WebAssembly never really took off,
Starting point is 00:20:15 but the use cases of WebAssembly are impossible to get rid of at this point. There's so many things you actually have to bring into the browser, and Rust can do that. I try that with Go. It's not a great experience because of the garbage collector and some of the complications that come with running Go in the browser. So you wouldn't do that, but with rust you can, right? So I think you kind of have to approach this from a perspective
Starting point is 00:20:39 of like, what am I doing here? Like what's the problem that I have? And then you said, like, I want to have not more than three, four technologies in the beginning, what they use it with. And then finally, I guess the one that we didn't talk about, which is pop the Java strip slash tie strip the ecosystem. Well, in the browser, you can't get rid of it. On the server, I think there's, I'm very comfortable.
Starting point is 00:21:01 conflicted on this because I actually think that it's a pretty good language and environment at this point. Particularly, like hypothetically, if someone were to make the JavaScript 2, like the Python 3 of it, and get rid of some of the naughty stuff. Like, I don't think anyone should be doing this because I've seen the migration. But there's a lot of good stuff in the language now. That's just also some really old dumb stuff. Why am I not building backend services with TypeScript right now? Actually, it's because of the NPM ecosystem.
Starting point is 00:21:27 Like, that ruins it for me. I'm a low dependency kind of guy. I want to control my shit. And it's just, I feel like it's impossible for me to build productive in the JavaScript ecosystem with under 500 dependencies. And that makes me uneasy. On the browser, I can live with it because I don't have much choice. But on the server, they really have to do it.
Starting point is 00:21:52 Do I have to deal with the fear of all of it and managing it? So it's like, that honestly ruins it for me. more than anything else. And so maybe that will change as that the ecosystem matures in other ways, but that really, I think, is the biggest reason.
Starting point is 00:22:08 And it doesn't have enough benefits for me on the server that I wouldn't just say. Like, there's this idea that you have a unified codebase. And every time I'm trying to make a unified codebase work, I just realize that the browser
Starting point is 00:22:19 and that the server actually is sufficiently different. It's very hard to actually have the unified codebase. And so then it just be explicit, like do some open API and again, some code channel.
Starting point is 00:22:28 and it actually feels much better. Yeah, the biggest argument I keep hearing from developers for TypeScript is the unified code base, that you can have a React in TypeScript and then on, you use, let's say, Node, because that's or Express or whatever, on the back in and now you have the same language and people can contribute, which I think was very, very compelling, especially before we had AI tools, because now you didn't have to learn a new language. Now it might change. I said this recently in a sort of talk I gave it a meetup, which is I think right now the floor is raising, the ceiling, not really,
Starting point is 00:23:04 in the sense that the expectations that everybody has in everything is much higher than it was before, but also because the expectations are higher and more tooling to enable it. And so code generation, for instance, right now, is in a much better state than it was even just two to three years ago. You can even buy off-the-shelf open API to SDK generators now. Right? It's like, I don't even know what the companies are called.
Starting point is 00:23:27 but stainless, for instance, is one of them, which just generates all the APIs for all the AI cloud providers right now. Yeah, for SDKs, yeah, there won't them, yeah. That was a huge cost of doing anything at Century. It's like, how do you keep these SDKs around? It's nice, like, well, I have an open API shenan again. Somehow magically on GitHub and SDK appears,
Starting point is 00:23:47 is it then really so important for me to have a unified code base if the code generation is so damn good? Like, all of a sudden, it doesn't matter quite as much anymore. So I think that, There is definitely some value to it, but you also have to imagine that if you have the unified codebase, then the boundaries are sort of accidentally very undefined, particularly with RSC, I noticed this a lot,
Starting point is 00:24:08 where it's very hard to actually reason your head around what's on the server, what's not. So actually having this extra boundary, for me, is almost like a feature. Speaking about AI, agentic coding, one thing that was very interesting that you mentioned is there's two of you in the startup right now, and you have just a lot of, of kind of, you know, like these AI interns running around, cloth code, codex.
Starting point is 00:24:32 How are you using them? What are you seeing for them? And by the way, can you all share what you're building with them? Speaking of how Armin and his co-founder use an army of AI interns shows just how much software engineering is changing. Our work as a software injury is no longer sequentially doing one thing after the other, but we can kick off parallel work with AI agents. This is one of the biggest changes in the software industry in a long time. A leading example of this change comes from our season sponsor, Linear. Their cursor integration lets you add cursor as an agent to your workspace. This agent that works alongside you and your team to make code changes and answer questions.
Starting point is 00:25:07 Say you have an issue in linear, like a buck fix or a small feature request. You just mention cursor and ask to take a look at it. Or you can simply assign the cursor agent to a ticket and follow along its progress. This delegates the issue and off cursor goes. Cursor then runs in the background, writes the necessary code, opens a pull request, and once you root the changes, you can merge it. It might not always one-shot tasks, but it's pretty good, mostly because the agent had access to context that's already in linear,
Starting point is 00:25:33 project plans, backlogs, issue descriptions, discussions. It's so useful. You can now delegate buck fixes, trivial work, or tedious tasks to the agent, then come back when it finishes. Linear sort of feels like ground zero. You've got to try it out. Head to linear.com. Yeah, so we scoped out email to be in fascinating space to work in,
Starting point is 00:25:53 and so you can kind of imagine what the kind of problems are that you have when you work with emails in one form another. And maybe a little bit of why we went into email is because one of the things that there's very evident now of the biggest shift in at least my lifetime, what's capable of computers, is that natural language processing is now available at scale
Starting point is 00:26:18 in many languages for a reasonable amount of money. and that one is a good reason to go into email because email is all about natural language and there's a lot of good data in it but it has been very hard to work with the data in the past. If you read my blog, you know that my opinion on AI stuff for programming was incredibly negative
Starting point is 00:26:36 up until February, March, where it slowly moved around and then in May I was like, holy shit, nothing is going to be the same anymore. And one big reason why I'm actually so bullish on this is because when you think into you, and actually it doesn't really work as well for teams today. I think. Maybe there are ways
Starting point is 00:26:54 and I'm going to see as I'm scaling out this company like how it's going to work at scale. But one of the reasons why I think it will actually be really good is because if I look over the years of all the stuff that I wish I would have been able to do, just didn't have the time for it. And I had to make this explicit decision not to do it. I have too many cases of this, right?
Starting point is 00:27:14 I remember we were reworking the grouping algorithm at Century. The grooving algorithm? What is that? It's like making sure that two errors, which are very, very similar, are grouped into the same group. A grouping, yeah. It's like it's a data problem at scale,
Starting point is 00:27:30 but it's also one of very specific data. And so we spend like three weeks building just a basic visualizer of ourselves to figure out if the new algorithm works better than the old one. But today... I know for a fact that I get this tool in 30 minutes on the site from Claude.
Starting point is 00:27:44 It's around 5,000 lines of code and it looks better and has like pretty UI and everything. Right? So there are many projects that didn't go anywhere because the legwork needed to build yourself this bespoke tooling to even be able to pull this project off like a migration system or something like this. It's just too much. Right. And so that's, I think, one of the big changes that I'm noticing for myself is that I have so many better tools around now that just make it much better to work with. I have like I'm at the moment I move from Terraform to Pulumi for one of the services, just.
Starting point is 00:28:19 to see how it goes. And I had Claude build me, like, my perfect control system to get my logs and visualize what's going on in production. And I would never have done this before, just because, like, it wouldn't have worked. Not a big question is like, how about this work on a main codebase, right? My co-founder is not a technical co-founder. But that doesn't stop him from using Claude or Codex now. And it's actually great because it means that even someone who maybe,
Starting point is 00:28:49 you normally like in certain stages of the company there's a bunch of stuff where you wish you could do something but but it actually requires maybe more coding skills than otherwise right and so there's an entire code base where I basically don't look at which actually validates a core part of the feature that we're working on and it's just vibes and it works it's not pretty but it doesn't it's not necessary because like the whole point of it is figure out like is this the experience that we want and and this is great and then the thing is like how much of the code that I
Starting point is 00:29:19 writing, which I feel like this is going to be a foundation of what we're doing, is at this point authentic. And like, I guarantee you it's more than 80%. And it's like there's very, there's very delivered code which is not written this way or like where we put a lot of effort into it. But most of the code that exists in this code base is not so important. It's a bunch of API endpoints. It's a bunch of open API specifications. It's a bunch of run off the mill pieces of code, they should look nice, they should pass all the tests, they should follow exactly the architecture that they want. But if you were to compress them down and say like, what's the actual information contained within them is very little, right? And so with the right approach,
Starting point is 00:30:05 you're actually very productive activities. What made you so negative or like just like pretty negative up to around February or so, the things that you were just really skeptical about? And then why have you become so much more positive about it? like these AI coding tools or agentic coding, right? The biggest thing is that it actually now starts doing work that I hated doing, but I know I had to do. So that is like, as an example yesterday, I really had to figure out why one of like the endpoints that I have work running there
Starting point is 00:30:39 in production didn't work quite as it should. And it was not a problem of the error in the code. It was a problem of EWS permission. And there was a chain of three things that it is wrong. That all resulted in it couldn't do what it was supposed to be doing. First, there was an IM permissioning error. And then there was a wide listing error in a different system. And then the final fuck up was, I forgot what it was.
Starting point is 00:31:09 I think, but the final fuck up was also something with maybe that was actually logic. But the point is it was one error and one error and another error. And you couldn't see them all at once. No. And I was also busy doing something else and I was looking at it's like, I know I need to debug this. But if I'm going to do this now, it's going to take me at least two hours.
Starting point is 00:31:26 But it didn't matter because Claude was doing a lot of it. Like I still had to copy paste a bunch of things from the logs. But it knows so much because it has world knowledge access. It can combine a bunch of things together. And I could make progress on the thing I was doing while it was also fixing and debugging this production issue. And that is, There's a significant improvement.
Starting point is 00:31:50 The same thing with one of the things I hated the most was creating repo cases. But I know that every time I got the good repro case, I enjoyed it so much more. And now I can say, Claude, make me the repo case for this. This is roughly what it looks like. Let's not try to fix it. I figure I was like, how do we get this one function that I can keep calling in a loop? That is what we're trying to figure out here, right? So I was not able to get that experience out of cursor, even with the agentic mode.
Starting point is 00:32:18 prior to to May, April. But that was really what changed my opinion on it. And then as I started going deeper and deeper
Starting point is 00:32:31 into, okay, now that I know that I can do this and I can do this and I can do this, I also got incredibly adventurous in giving it more things to do.
Starting point is 00:32:40 And now I can, I have a new sense of where it could go just by, giving it the things I don't want to do, but staying in control of the things I do want to do. And that was the big shift for me, was recognizing that it can do that in a way.
Starting point is 00:33:00 And so far, what are you seeing and what these agenic tools change about software engineering and what are the things that don't change? Especially, you know, you've been writing code for like, what, 20 years or so? So system architecture and like overall complexity and like how you build.
Starting point is 00:33:18 things in a way that something stays maintainable as it scales into a larger and larger company. There is a lot of experience in it. And I also didn't have all of that experience at any point in time. So I feel like, this is one of the things where I really like as time went on, I learned a lot more, which gives me a benefit over my prior self that didn't have that. That is going to continue to be true. Now, there's an argument to be made that maybe Claude is like, systems like this will help you and maybe
Starting point is 00:33:50 can even bootstrap these sort of things around it. If you fully delegate everything that you're doing to a machine, then the person that doesn't do that has a match on you. Right? Because like there's going to be some innovation that is not in these things yet. Right. So like right now what these systems spit out is sort of our best understanding of how stuff worked up to this point.
Starting point is 00:34:11 Right. And so having something that maybe at scale is not in these corpuses yet is where. some of the edge comes from. It also creates a better culture in a company if you're not completely delegating everything to a machine. Right? It's like,
Starting point is 00:34:25 because like what's the point if it's just a bunch of like this? Like this is one of the reasons I want to hire more people now. It's just because Claude is not a human. And it's just that there's a, there's an energy to a well-running company and to, to motivated people that's just a computer can't replace. So I think that is,
Starting point is 00:34:43 that is a big part that sticks with you as an engineer is like understanding how to create systems. Understanding when the code that it produces is is more right versus more wrong. So I don't think much of this has changed. I feel the same programming with Claude, even if I physically write less code. It's just that I don't know that the physical act of punching the keyboard is not quite the same anymore. What is different though? What has changed with it? I think the cost of building your custom tools went down. the ability to make better decisions now for me is heavily improved
Starting point is 00:35:24 in parts it is because it Googles very well it searches the internet very well right it's like if I now have a problem of a of a type that I didn't have before it can teach me this problem as this is one of the things I used a lot with is like I don't actually want to get the answer I also want to, I also want to understand why that's the answer. And it can do that and it can put it into the perspective of it. I'm not very good with a lot of very complex math. And I found that every once in a while I need to do that.
Starting point is 00:35:57 And having Chachipity dumb it down for me helps me a lot. It's like it, it's seemingly certain, if a problem is presented in a certain way, it works very well for me. If the problem is presented in a different way, but taking a very abstract paper, way in which you have it like in many papers. I do very, I'm not nearly as good as I should be, I think, for what I'm trying to do.
Starting point is 00:36:21 But Chachypity helps me grasp it quicker, right? So I think that's a huge change. The other thing that definitely has changed and I even wrote about this recently is that it brings a whole bunch of people into programming that didn't do this before. Yeah. Like I wrote about this before,
Starting point is 00:36:38 but like I had a four-hour conversation with an air traffic controller on a train who as a like talks about how he used chat chvdd and like how like he can even do some programming with it right it's like that's a program now yeah like he's a very specific problem he wouldn't have programmed before but because there's codex in the chat chdbdd subscription he's able to solve some problems that he wasn't able to solve before and that is a huge unlock right it's like we're going to have many more programmers just because as a byproduct of asking a question chat chdpddd is programming and and a human it's going to keep running that code and maybe it wants to do some changes to it. One of the side effects of Flask being so popular is that it has been in university courses. It has been in onboarding for new programmers in like boot camps and things like that. So I've been exposed in one way and another through how long it takes for someone to learn programming. That's not a thing that you do in a day.
Starting point is 00:37:34 So for you to be a productive program and it's capable of spitting out the program, that's a couple of months of work. And now someone can press a button and get something. And then maybe they enjoy programming for that, right? Maybe it's like, well, now I see the effect of having produced something. Like maybe he didn't do all the things, but like, but it's still the same gratification you get from it. And then maybe that's a way for you to, for people that wouldn't have gone into programming
Starting point is 00:37:57 to actually go into programming because the onboarding has been so much easier. We talked a lot about programming languages and you have very specific, you know, you have your opinions on them. But with AI and CloudCode, codex and all these more and more capable agents being here. How do you think the importance of programming languages will change? Do you think it'll be limited to a few people or just fewer and fewer people like yourself with others who actually have spent a lot of time understanding the pros and cons of programming
Starting point is 00:38:28 languages and thinking well beyond? Because it sounds like you can just tell AI to solve it and whatever, give it an existing codebase. It'll just do it, right? So there is an argument to be made that programming languages might not be that important anymore. What is your take? Well, I think I will be consistent with the prior thing that I said,
Starting point is 00:38:46 which is I think I have a lot of respect for programming languages because of the trade-offs that are in there. And with AI now, I think you might have to start looking into different kind of trade-offs. Another reason why I ended up with Go in my startup is because I just noticed how good Go-code scales for AI because the abstractions are very thin. It understands the code better. So clearly there's some, I did the measurement.
Starting point is 00:39:12 I made it write a certain type of program in different languages 10 times and see how often did it pass. And it just noticed that it did so much better on Go than it did on Python. And much better than at Rust. Clearly languages matter because the quality of what the agent is going to do will matter. And then is any of the languages they have right now, is that the perfect language for both a human and the computer combined work environment? I don't know.
Starting point is 00:39:40 maybe found the pain of programming languages, but maybe that's also exactly the moment in time where someone will come and say like, hey, I actually have a brilliant idea of what the better trade-off is here because the cost of doing this kind of stuff is going down with AI, but the cause of this and this stuff is going up
Starting point is 00:39:55 because of AI. So let me rebalance this in a different way. So actually, I think that program language will continue to matter a lot, particularly the trade-offs that the program language implies on the runtime environment matters a lot more. So I don't think that's going to change,
Starting point is 00:40:10 I think it's just going to, everything accelerates. Everything is going so much quicker right now, or it feels to go so much quicker because the fundamentals are shifting around. And this is not just with, now everybody wants to build all the new things. It's also like you feel like if you're not working all the time now, you're missing out on some of the changes. And I don't think that's just on the product side. I think it's on the tech side too.
Starting point is 00:40:35 I'm pretty sure that a bunch of people who are feeling like, I want to be the first person to build the programming language. It works really well for agents and humans. But it's not just one person, probably a lot of them who are looking into this at the moment. It just feels so, like there is, in fact, a moment right now to build the perfect language or a better language that works in that environment,
Starting point is 00:40:59 because we are probably not going to get rid of chinticoding. And what is the likelihood that all of the languages that we have right now are the best ones? probably not that high, right? So it will be the same and different in many ways. And one thing not to be underestimated is that the likelihood that computers are just going to program and no human is going to be in a loop, I don't think is very high. I think the human will stay in the loop longer than we want and in many more cases than we
Starting point is 00:41:26 want. And so you cannot go down to say, like, well, the optimal output is write a bunch of assembly code because then nobody can review this anymore, right? And so if anything, the trade-off will have to be like a higher level so that reviewing gets easier. Also, don't forget that the thing with assembly, right? Like, one of the reasons we have higher language languages is assembly needs to be different for every architecture type or every CPU type, which is why, you know, like Java was so popular in the early 90s. It ran on both Macs. But if I can't just have the AI ported to my auto operating system, they could make the argument that maybe that is not needed anymore.
Starting point is 00:42:04 For my first word, I don't think that's the opinion that I hold. Yeah. But it changes the cost of some of those things, right? Yeah. But you mentioned an interesting thing, which is how everyone seems to just be working all the time, which is this interesting contraction, how AI or these ages you can kick off and you could, in theory, go to sleep and have it just run in loops for you. And so you could work less. But in fact, we're seeing working more.
Starting point is 00:42:28 And we now have this thing, which is very amusing to observe that AI startups in particular, are increasing demanding or advertising this 9-96 from 9 a.m. to 9 p.m. 6 days a week, especially in NSF, they're putting it in the job adverse. They're posting it can probably online that in the office at midnight or after midnight people are still there and working hard. You share some thoughts on this as well on your blog. It costs quite a bit of stir. But what is your take on this? Because you did say, you know, like there is a lot of energy right now. there's a feeling of not wanting to miss down. Clearly, that's why these people are doing it.
Starting point is 00:43:08 Yeah. So a handful of thoughts, one of which is, I actually have to credit Peter Steinberger quite a lot of pushing me into a chant decoding because he worked in a comic called PSPDF kit, which is sold. And then, I don't want to tell his story, but long story short, at one point, he was kind of stopped programming and then he fell in and said like, hey, I found this this computer which programs for me
Starting point is 00:43:35 and he said like he doesn't sleep anymore. He's like I'm so addicted to it. You should try this in a way. Like he didn't quite say it like this but I'm skeptical right. But the computer that does your work, you cannot sleep because of that's kind of so amusing. Yeah, it's like I realized like
Starting point is 00:43:51 it does something with your brain if you do this as a chinticoding thing because like in the beginning I felt like any minute I'm not having it running and doing something. I'm just wasting time right now. It's like it was a huge part about this. It's almost like a drug, right? It has this instant gratification of something happened and you can kick it off again and kick it off again. It's like this, it is a lot machine, right? And so I think I attribute some of it to that. And it took me a while actually to come
Starting point is 00:44:18 out of this way of thinking because it's not very sustainable. Like there were a bunch of nights where I just, I did through the night with a chint decoding, not because I felt like I'm incredibly productive when something using from a startup. It's just like, this is, like, this is blowing my mind. I just, just one more prompt. One more prompt. So I think, like, to some degree,
Starting point is 00:44:39 because if you're so AI close, that probably also contributes. The other one is, I think everybody sees the change and wants to do as much as possible. I think the tricky bit with the 9 and 6 thing is that 9 and 6 defines a very, very specific kind of work regimen,
Starting point is 00:44:54 which is you work 12 hours a day and you basically don't have a weekend. No. I never did something like that. even though I probably worked sometimes 80 hours weeks. And the reason for this is that I know that I cannot be productive all the time. If I work late at night, I will shoot in the morning. I have three kids.
Starting point is 00:45:15 I have a wife. Family is very important for me. It's the most important thing. So I will always arrange my work around that. You can work with intensity, but you can still do it in a way that is not, crazy hours at crazy times, which is like you're optimising to the optimal output that you can produce somehow.
Starting point is 00:45:38 And I also want to say there's a huge difference between if you are the founder or if you're a very, very early engineer with a really good compensation system where equity means it's something to you from most of the people who join a company later on or where the company has no realistic path
Starting point is 00:45:56 of their equity being worth something. There's absolutely no point in putting a ton of effort into something where only one person benefits from. And there's a lot of like subtlety to all of this. And what annoys me with the nine and six point in particular is that they pick that word, which if you go back to why that naming even exists
Starting point is 00:46:21 is because of basically there's no subtlety to it. It's like you work 12 hours a day, six days a week, and you give up what is otherwise in your life. And I don't think that's, I don't think it's a tradeoff that anyone should make it at any point in time because there is more to life
Starting point is 00:46:43 than working in a company. And that is not the thing that you should be putting on your flag, which has such a negative connotation because of exactly what it did. Well, people literally killed themselves over it, right? And like if you want to be a high-intensity work environment, then say that. But but but but be more because in practical terms, you cannot be a company that runs like that.
Starting point is 00:47:10 You can only be a company where high energy people work and to build themselves up. But there's a limit to which you can do it. And you have to be transparent about what the cost of this is. And I've been in the industry long enough that I've seen many, many, many bad outcomes. And in particular, through my community, a lot of, of people even went down, like psychosis and schizophrenia and a bunch of things, where your likelihood of going down that also increases during certain years with unhealthy behavior. Now, having worked at Sanchu, which is all about error handling or at core, is there, you've
Starting point is 00:47:45 probably seen tons of stats on most common errors that are coming up in different servers. What have you learned about error handling and how to build better software? with fewer errors from a developer's perspective. So the biggest thing that I learned and ironically hasn't really dramatically shifted despite the existence of tools like Century and their popularity is that many of the ways in which errors are worked with just doesn't carry enough information or only carries useful information for debugging in debug builds. and that's problematic
Starting point is 00:48:29 because the most interesting errors don't happen in debug, right? Now, maybe this is changing a little bit with AI because like the cost of producing a reproducer from not perfect information might go down. So maybe there's a countermove to that, but I really felt that as a language designer, as a VM creator, you should put a lot more emphasis on making sure that the errors can carry really useful information rather cheaply,
Starting point is 00:49:03 even in reduce and in production runs. And so not a lot of ecosystems don't do that, which is very sad. And I think that the fact that Python had such great introspectability without further, like Python is a slow language. So if you already throw all your debugging tools at it, it doesn't get significantly slower. That actually was the reason. existed because Sanchi was able to show local variables for Python. It had all this rich, powerful data that for many languages even today, we're not able to
Starting point is 00:49:34 do because the runtime doesn't support it or the runtime effect of doing that is too high. And so that's the thing that I learned the most. It's like your experience of debugging issues is so much better if you have rich, which information errors. And unfortunately, both language languages, language creators run times often neglect errors so they don't carry the right information
Starting point is 00:50:01 and the application and library developers often don't think about errors at all or they capture them down in the wrong places and then the useful stack traces gone all that sort of stuff because you think they're exceptional they're not happening all that often but unfortunately every time
Starting point is 00:50:18 did it happen you know you didn't have enough data so just in general like the design of errors is completely, it's not where it should be given how important they are. So that's one thing, but the sentry works across a bunch of different languages, almost all languages right these days,
Starting point is 00:50:37 or at least the ones that are widely used. Have you seen interesting patterns between the types of errors or the frequency of errors of using language? Basically, I'm trying to get to, like, how important is it to choose your language in terms of, you know, like, having more correct programs.
Starting point is 00:50:55 Well, different languages crash in different ways and not just because of the language but also what you built with it. And so, for instance, JavaScript, at scale, like if you look at a lot of websites,
Starting point is 00:51:10 errors all the time. The percentage of errors in JavaScript that actually are like meaningful is very, very low. Yeah, like whenever I open up my deep, you know, developer tools and then it shows you the errors and warnings
Starting point is 00:51:25 and like every single website or almost every. And that makes sense because you very few errors that you can actually cause a JavaScript and crash your browser tap for instance. Right. So it's like it's like on error resume next. You're hobbling along. Like something is broken,
Starting point is 00:51:41 but doesn't really like, but like the existence of an error in a console lock doesn't imply that the website is broken. But for instance, like it might like now the evently son might not fire anymore. So the click is a dead click. which is something that Century needed a session replay product to find because the actual error that you would find in Century
Starting point is 00:51:58 was insufficiently linked to user not being advanced on the page. On the other hand, if you have a computer game, it's written C++ and it crashes, your session is over, right? And so they're comparative fewer crashes in computer games compared to all the nonsense going in JavaScript, but when they do happen, they're much more meaningful. And so it's very hard to say like error rates in different languages and the forth,
Starting point is 00:52:23 because the impact that these errors have is very, very different. So C++ code crashes in computer games, shockingly little. Like, the amount of traffic that sentry gets from these is very, very low. But the usefulness of each individual crash report in C++ bus is so much higher.
Starting point is 00:52:38 So it's just, it's a very complex topic at scale to say, like, how does something error? Because it really depends on, like, what does it take down? Obviously, there are certain types of errors that if you see them long enough, you feel like they really shouldn't happen. To some degree, there was a large-scale realization
Starting point is 00:52:56 in JavaScript ecosystem that type checkers could get rid of a whole bunch of class of errors because at the very least you have to explicitly check if this thing is nullable or not. But I also didn't ever get the feeling that the adoption of TypeScript would dramatically change anything about, like, centuries JavaScript error rates, if anything,
Starting point is 00:53:14 none of that adoption had any meaningful impact on how crashy, yeah. Well, because what I would have expected, right? Like we know, and I think it's pretty commonly assumed. I'm not sure if it's proven that typesafe languages will reduce certain kind of more obvious errors. TypeScript does this with JavaScript. I mean, it has a compiler, but in the end, you do get a check before it all gets compiled to JavaScript. So we should have fewer.
Starting point is 00:53:46 That's what I think. So you're saying that you didn't really see too much change. No, I see if there was an impact, it's unmeasurable. And of course you have, like, you have other sort of frequency functions that might happen that offset. But like, maybe whatever improvement that you have on, like, catching, like, some of those, maybe that sort of makes you just more adventurous to build more complex code in comparison, right?
Starting point is 00:54:12 A lot of the time of Century was the ZERB era of crazy complexity. And so then, like, many of the errors were, related to not like, did I, was this nullable or not, but now to a sudden it's a microservice that I'm talking to and they have misaligned versions and my type check actually didn't help me at all because someone decided that on a network layer, it's now null anyways, right? So the increased complexity of many of the software that's actually dealt with over the years, I think you could have probably measured that, especially I remember the adoption of, of, of, more and more complex things in a React ecosystem
Starting point is 00:54:53 just dramatically increased the types of errors. Like hydration errors were not a thing for many years. And then all of a sudden, there's a whole class of new errors coming in because now all of a sudden, the view stability between initial server render and what the JavaScript dynamically loaded on time does. It's like that was a whole class of errors that didn't exist before and all of a sudden does exist, right?
Starting point is 00:55:16 So it's very hard to measure these things just because of how we change. change and how much more complex our apps are. Yeah, so, so, so like you're saying that's true, like, before React and, you know, like hydration errors are specific to React, but those, that whole error category just didn't exist.
Starting point is 00:55:32 And then as React spread, they're kind of pretty common in React. So now you get, technically, you get more errors. It turns out to be a very safe business to be an errors because they're not really going down. Yeah. Well, and then I think we're going to extrapolate it obviously with the AI
Starting point is 00:55:48 tools using, typically using the most popular frameworks or the one that has the most training data, which is the most popular ones. And we'll likely see an explosion of code and deployments. So yeah, it's probably a good business. But having been in the errors business, how did you change your or at the team that's century? How did you change your approach to building software with errors in mind?
Starting point is 00:56:14 Were you more aware of it? Did anything change at all? Because it's interesting. you've had such good insight into like all the I'm going to say mistakes they might not be mistakes but we can say you know error is generally not a positive thing but you've seen all these things come through so did you change how we develop software? Yeah you would think that working in observability company makes a great at building observable products and and the reality is that there is a reason errors aren't off the thought right? It's like you you almost have to force yourself to take care of this five. percent case or 1% case, right? And that's a continued problem. Like every pull request in a way, you are now like, okay, now it passes.
Starting point is 00:56:58 So we have dealt with this. Like, are you going to put the extra time in to also make sure that even if it like now fails that it reports the error correctly? I don't think that Centri's code is more correct for error reporting than any random other code out there is because the act of taking care of good error reporting is a deliberate act. The act of, like, good logging is a deliberate act. The act of good metric
Starting point is 00:57:23 supporting is a deliberate act. And you will train that muscle a little bit over time. But I think, like, you maybe get 50% better, 100% better at it, which is still very, very low. We should get, like, a thousand percent better. Like, it's, like, that's where we would see the, like, the improvements.
Starting point is 00:57:39 Like, if everything, like, if 90% of errors are really, like, error cases are really well reported, like, that would be really good. But, like, That's a really massive lift. And you would only really get this lift if you stop programming like we are. And maybe the languages themselves have like pulling this into every stack frame into every sort of corner case. So I think that the real lift would come if a bunch of people that now build these observability companies.
Starting point is 00:58:06 So like, okay, now that we know what our problem is, which is data collection in the first place, how do we build a programming environment that makes this native, right? For many, many years, the biggest challenge for just bringing awareness of what we're doing was context locals. What is context local? There's an excellent talk by Ron Pressler who wrote the, I think the original implementation of virtual threats for Java, who later worked on the Loom project to bring virtual threats straight into Java. basically made a strong argument for a stacked frame. Like, what's so great about stack frames?
Starting point is 00:58:48 Like, stack frames are amazing because they carry information basically implicitly with it. And he should have demonstrated what you give up, if you move to Asyncowade, where you basically chain together a bunch of promises to each other. And the immediate problem that we have,
Starting point is 00:59:05 if you start moving towards promises, is that now it's very hard to carry through information through all of the potential paths that this promised chain would take you. So for instance, if you want to say, like I have a correlation ID, which should be attached to every single log event,
Starting point is 00:59:21 it should be attached to every single error event, that might happen as a result of this original thing I'm doing. With stack frames, super trivial. It's sitting somewhere in a stack. I will not lose it. I can walk up the stack. I find it again. But with promises, now I need to have this chain together.
Starting point is 00:59:36 And so context locals is what is called in some languages. In dot net, it's called the, execution context. It's basically a hidden piece of almost like a fret local, but a much more narrow scoped one, which flows with the logical flow of execution. And why is this necessary?
Starting point is 00:59:54 Because if you do something like open telemetry, if you do tracing, if you do things like you want to have a correlation idea, you want to have it show up everywhere, right? And for many years a century, it was like just talking to people like, hey, you know, it would be really great to have context locals everywhere.
Starting point is 01:00:08 And JavaScript, I think in a browser still doesn't have them. in the backend it eventually got domains initially, then it got async hooks, and nowadays it's called async local storage. It sounds like to me, like as you said, like as an engineering team, let's say even at Century, you kind of your focus is to get the thing done,
Starting point is 01:00:26 you know, you test that it works correctly. And then when it comes to like errors, as you said, it's kind of usually afterthought either when it blows up or, you know, if you've got a lot of time or if you really need to make it sure that it's going to work correctly. but with languages programming languages it almost sounds ironically that in many languages it's an effort thought because as you said if you thought it through better saying hey at some point the program will crash or there will be an error what can I do and you know there's a context local thing that you talked about I don't think it's an afterthought I think it's deliberately not done because you have to imagine like there they're there there are different kind of forces in a programming language that say like what should the language do and some of them are very clearly odds with each other so what's the problem of context locals? Well, they make every call slower, right?
Starting point is 01:01:14 So now it's like all the people they want a faster language. They're like, well, now you've got to make a really strong argument why you should be doing this. I remember one of the biggest fights over the years that sort of was hidden behind the scenes was
Starting point is 01:01:27 on native platforms. Compilers at one point made a decision that we can get one extra register by basically using the, it's called the stack register. You would basically, you would give up the last register to recover the stack frame for one extra register for other users, and then you would have to use a very complex dwarf unvinding system to recover this, right? So you basically say,
Starting point is 01:01:52 like, we're moving some of the complexity of being able to just at runtime walk for the stack to doing it delayed through a very complicated binary program, which is embedded in this debug information file. But what do you lose through this? Well, you lose runtime profiling capabilities, because that's too slow to do at scale, so I can't do an in-process profiler, which turns out to be incredibly useful because you will find information that you wouldn't find through tracing, right?
Starting point is 01:02:20 You can do like, basically, you can run a sampling profiler. It also made it incredibly hard for Century to do proper stack reports for native code because I might not be able to recover the debug symbol because it's a binary driver from someone and nobody ever gave me the DLLs. Yeah, you don't have a source code or the mappings.
Starting point is 01:02:40 I know, like, Facebook, for instance, Facebook's app secretly uploaded Android system symbols all the time. Like, your device were randomly assigned if it should upload it or not. And they were harvesting all the system symbols to increase the quality of the error reporting. Like, if you have Facebook scale, they can do that. Like, Century couldn't do that. Like, we can't just magically have every Century SDK just upload a bunch of files that would just destroy user trust. But eventually, anything that only took place two years ago,
Starting point is 01:03:08 someone at, I think it was Debian or Redhead was like, well, this is nonsense. We're going to change this now. We're going to bring the stack pointer back. And it was a fight. It was a fight to do it. And it was like, well, we bench ducked all of it, but just losing 5%. Except for Python, where the Python interpreter, for whatever reason, got like 20% slower. So they didn't do it for Python.
Starting point is 01:03:28 But it was one flag, right? It's one flag. But what matters is that flag is set by default to everything, right? So it should be on by default. But this 5% of performance difference, which was assumed to be more than that, there was a year-long pressuring, right? So I think it's hard to do because you have these different interests in it. And some people want the performance and some people want the debugability. And they might live in completely different worlds.
Starting point is 01:03:58 Like maybe there's one person that really cares about performance just never has to look at any crashes because that's not the area of responsibility. And then you have someone who maintains a fleet of comprehensive. services and every one of them has a small percentage of crashing and often doesn't run the code that they have and then once they have to crash they don't have the data that they need. This is so interesting because it just reminds you of how difficult it must be to design a language because again you have so many users and as you say the person who just had a crash they actually want ideally the language of support to get that back in a very easy way. But now it will by default slow it down either to perform.
Starting point is 01:04:37 or through higher memory usage. You cannot have a free lunch, right? Yeah, it looks like the lunch comes with some price. But look, I have a lot more respect now, I think, for language designers, just because of, like, especially if you start a program, you have this naive idea of like it has to work this way, right? And then you, and then over time you realize, oh, there are all these tradeoffs you have to make.
Starting point is 01:04:59 And some of them are really hard to quantify, and you might even have to make some very unpopular decisions. And there is no real right or wrong. That's just a lot of tradeoffs. maybe this is also why we have so many languages to some degree, but it is very fascinating to think about what some of these seemingly little defaults have in terms of what you can do with it. And then maybe what kind of people are going to pick your language.
Starting point is 01:05:22 I thought Python was the most amazing language ever for building web services because I could inspect every process. And it didn't make it any slower. But now I know that's also why Python was slow to begin with, right? What are some things that you've learned being an early engineer, almost like a founding engineer at a startup? And what is advice that you would give to an engineer joining a startup, a fast-growing startup right now early on? Well, I can give the advice from someone who would have joined a company 10 years ago, right? Look, the situation changes all the time, I think.
Starting point is 01:05:57 But my life in many ways is a very linear path. where it's like there's an opportunity I make a decision if I take it or not. It's like the secretary problem but I'm kind of taking not too many attempts of it right which is like I just want to like the passing bar for me
Starting point is 01:06:13 is do I think that this is great do I want to work on it and then maybe I take the opportunity to work on it. And for me in particular I'm shaped like a person that is actually not a great employee because I don't necessarily
Starting point is 01:06:27 go and fit a particular mold my titles at Century in many ways over the years were I didn't even associate with them in a way, right? It's like at one point it said, I don't know, software engineer on it, but I was like responsible for an office here with people underneath where I sometimes felt like I'm a payroll provider, right? Like there were so many other things in it because like a small startup has this many, like now someone needs to deal with this stuff too, right?
Starting point is 01:06:58 Like you need to hire someone to even just bring them things like this furniture into the place. If you join a company early, everything is in flux, nothing is well defined. And some people like that. Some people don't like that. And I think you have to be just willing to experience new things if you start at a scrappy startup that particularly maybe comes from someone who hasn't founded a company before, hasn't been there when a company sort of grew from small to big. Because even now for me, I just realize, and I kind of forgot, I need to do all of this and this again. But there are people that like that and people don't like that.
Starting point is 01:07:38 And you just have to be aware of the consequence of this. You have to path the way for yourself somehow, either by being okay with just doing whatever comes up or being incredibly ambitious and deliberately taking certain paths towards where you want to go, even if it might temporarily set you back in one form another. And now you're the one who started a startup as a software engineer. You founded your startup with your co-founder.
Starting point is 01:08:07 So far, how is it going? And how is it different to what you've expected, especially that you were on the other side, right? You were an early employee inside a startup. You've seen a lot of things. Is it kind of like, you know, exactly as you expected or is it just a different feeling? In some ways, I postponed the founding of the company many times, right? In my mind, at least, my recollection of early century was like, well, there could have been a path where we didn't do century.
Starting point is 01:08:33 We did something else, right? And then after almost like four or five years already, like I remember like when my first, like where all my shares wasted, I was like, hmm, is there a path where because there was also Vassel was coming out at one point where it took next chairs and built an entire business around. I was like, hey, is there a path where you could take actually something like flask and. and build a thing around it. So in some ways, there was like a multi-year preparation for me. So now it doesn't feel like, oh, it's actually different. It's just like, okay, that's the next logical step. Kind of, I wouldn't call them missteps,
Starting point is 01:09:09 but knowing some of the decision we have made at Century, I can now look at them and sort of look them from a different perspective. I have different opinions now about the equity, about the consequences of equity, how to structure certain things. Like this time around, basically, I didn't start with, here's an idea for like a product I want to build. The reason I ended up starting this company of a co-founder is like we actually sat down and said like,
Starting point is 01:09:34 hey, what do we want a company to be like? Why are we even in here for it? We already know that we like, we're the kind of people that want to start a company, but it's just like that's not an end on itself, like that something should come out of it, right? And so I didn't do that last time. It just rolled into it. Now you can
Starting point is 01:09:51 overthink it obviously, right? But it's also, there were many moments over the years that century where I felt like why I actually would have been great for us
Starting point is 01:09:59 to think about some of those things so now I'm thinking about them in the beginning. The experience primarily puts you there. I don't think you can sort of start out
Starting point is 01:10:06 and say like I'm going to do this. Obviously you can read a bunch of like stories of people that started startups and then you're listening to other podcasts and everything and you hopefully get a better sense of it
Starting point is 01:10:15 and maybe some to some degree over prepared but having gone through it yourself but just hearing from somewhere like I read a lot of stuff about startups even 2015. Right. I was sort of addicted to that thing.
Starting point is 01:10:29 And still it feels very different now doing it versus if I would have done it five years ago, just because of my own inner, the experience, sort of like how I feel about all of this, which when I have a level of urgency versus where I just sit it out. Like the experience kind of puts you in a different position I would have done otherwise. I don't know if that's helpful, but that's, I think, mostly how I feel.
Starting point is 01:10:54 Well, I mean, it sounds like you can like, you know, get all the opinions, read about it, listen to stories, but it'll be different when you do it in the end. And it's hard to sell what ways until you do it. Yeah, and it worked for some people, but I wouldn't have worked for me. Like no amount of reading of anything would have sufficiently prepared me. Some people it might, right? Like particularly if you have well educated, like a person who grew up with the right values to have this intrinsically sort of put on to you. But that was not my education, right? My education was. like just standard schooling, a bunch of university, and then there's a whole world that I haven't been exposed to that sort of I had to learn in one form another. And then just to close with some rapid questions, so I'll just ask and then you tell me what comes up. Of the programming languages, which one is your favorite?
Starting point is 01:11:43 And I'm really interested to hear. It's Python and two answers for why. One, because it gave me the career that I had. And so there's obviously a lot of emotional attachment to it. But also it is a bad programming language for many, many aspects of like misdesigned and whatever. But it is incredibly pragmatic. And I just like that. And I kind of want to point towards many of the things that Cal Henderson did over the years.
Starting point is 01:12:11 He was the CTO at Flickr originally and then Slack. Ph.B guy, right? But the pragmatism in which he used Ph.B. was in many ways of pragmatism which I always like building product with Python and it's like you don't really care if it's good or bad and that sounds like what you can do with it
Starting point is 01:12:30 and I could do a lot of Python and I appreciate it for this. And then what's a tool that you love using and what does it do? I will answer this as a not programmer. A tool I love using is a screwdriver. It screws. But the reason why I just love it so much and I think it's one of the things that I learned over the years
Starting point is 01:12:49 like I never really had a good electric screwdriver. And then when we bought our apartment, I just bought really good tools, including a screwdriver. And it has increased my willingness to drill holes and assemble furniture and everything. And that by far is probably the most favorite thing that I have now. It's just really, really good, well-manufactured screwdriver. I wonder if there's a metaphor there
Starting point is 01:13:18 that will apply to development as well. Like, you know, when you have good tools, you're actually a lot more motivated and willing and you're more adventurous as well. Well, thank you for this chat. This was really fun. It was really nice talking about it. One of the most interesting parts about talking with Armin
Starting point is 01:13:36 was how he went from being very negative about AI six months ago and now he's building a startup with AI agents like CloudCode and OpenAI Codex. I see Armin is a very pragmatic engineer. And if he went through this much of a change, after pushing himself to use these tools more, it tells me that it's worth giving AI coding tools some time and see how they work for you. For more observations on AI tooling trends,
Starting point is 01:13:58 check out deep-eyesing the Pragmatic Engineer, which are linked in the show notes. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating for the show. Thanks, and see you in the next one.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.