The Changelog: Software Development, Open Source - The best, worst codebase (Interview)

Episode Date: September 18, 2024

Jimmy Miller talks to us about his experience with a legacy codebase at his first job as a programmer. The codebase was massive, with hundreds of thousands of lines of C# and Visual Basic, and a database with over 1,000 columns. Let's just say Jimmy got into some stuff. There's even a Gilfoyle involved. This episode is all about his adventures while working there.

What's up friends friends? Welcome back. This is the Change Log. On this show, we talk to the hackers, the leaders, and those working on the best, worst codebases. Oh, my gosh. Today, we're joined by Jimmy Miller to discuss his experience working with a legacy codebase and his first job as a programmer. The codebase was massive, hundreds of thousands of lines of C sharp and visual basic and a database with over 1000 columns. Let's just say Jimmy got into some stuff.

There's even a gilfoyle involved. And today's episode is all about Jimmy's adventures while working there. A massive thank you to our friends and our partners over at fly.io. Yes, that's the home of changelog.com. Launch your apps, launch your databases, and even launch your AI near your users. Fly is the public cloud built for

developers who ship. Launch your app in five minutes at fly.io. Okay, let's do this. What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO, Dylan Fox. Dylan, tell me about Universal One. This is the newest, most powerful speech AI model to date. You released this recently. Tell me more. So Universal One is our flagship industry leading model for speech to text and various other speech understanding tasks. So it's about a year long effort that really is the culmination of like the years that we've spent building infrastructure and tooling at Assembly

to even train large-scale speech AI models. It was trained on about 12.5 million hours of voice data, multilingual, super wide range of domains and sources of audio data, so it's a super robust model. We're seeing developers use it for extremely high accuracy, low cost, super fast speech-to- text and speech understanding tasks within their products, within automations, within workflows that they're building at their companies or within their products. Very cool. So Dylan, one thing I love is this playground you have. You can go there, assemblyai.com slash playground, and you can just play around with all the things that is assembly. Is this the recommended path? Is this

the try before you buy experience? What can people do? Yeah. So our playground is a GUI experience over the API that's free. You can just go to it on our website, assemblyai.com slash playground. You drop in an audio file, you can talk to the playground and it's a way to, in a no code environment, interact with our models, interact with our API to see what our models and what our API can do without having to write any code. Then once you see what the models can do and you're ready to start building with the API, you can quickly transition to the API docs, start writing code, start integrating our SDKs into your code to start leveraging our models and all our tech via our SDKs instead. Okay. Constantly updated speech AI models at your fingertips. Well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for

free right there in the browser, or you can get started with a $50 credit at assemblyai.com slash practical AI. Again, that's assemblyai.com slash practical AI. Again, that's assemblyai.com slash practical AI. So we're joined today by Jimmy Miller, host of the Future of Coding podcast. Jimmy, you wrote the best, worst blog post. It was amazing. Nice little play on words there. Yeah, I recently wrote a blog post talking about my, my first job in programming and it's the best, worst code base I ever worked in. And so, yeah, it was a really fun post and kind of blew up. I mean, you know, way more than any blog post

I've written and got like primogen on YouTube, like doing a video on it. It was, it was really cool. Really neat to see top of the hacker news, all that stuff. It's really fun. Definitely resonated with me. In fact, I was stopping to check the technologies that you listed because I literally thought maybe I had been on the same code base. And then I was like, I was all C Sharp and VB.

I was like, okay, mine was over here in Ruby land. But I was just bringing up PTSD or something because I've been in a code base like this. Well, first of all, let's lay some groundwork here before I start to get into the details because this was your first job coming out of college. Is that right? I did not go to college.

Okay, coming out of schooling. Yeah, yeah. It was a couple years after high school. And successful business that you went to work for i assume yeah uh is a big credit card processing company that's now been merged or bought out or something but uh you know pretty pretty big company this was like a small branch of it that was like the customer support center but you know fairly in the grand scheme of things a medium-sized company but fairly large you know for actual software development right and so lots of money being made of course credit card processing a core piece of the world's infrastructure if you

getting fractions of a penny of every transaction i mean there's just a lot of money coming in. Money hides problems. The success just hides problems. This thing had so many problems, it's just hard to fathom that it operated. It was honestly pretty crazy. Going from the only code I had ever seen was code I wrote myself, to diving into this kind of code base

where we got hundreds of thousands of lines of C Sharp and VB with just crazy database configurations, all this wild stuff going on and just realizing like, oh, this is what real world code looks like. And of course, I had no other point of comparison. I now realize it was a little unique,

a little bit of craziness there. But yeah, it was one of those things where you think like, if you're a successful company, the code's got to be good. And I found out that that's not really the case. There's something about the naivety also of being fresh out of school or young to the industry. I think I told this story before, which I had a similar experience where I inherited some bad code, but I didn't have that perspective and just knowledge enough to realize it was bad code. I thought it was good code and I was a bad programmer.

And I probably, I mean, I was. But still, I gave it too much credit because I think this must be what good code looks like. It's so hard to understand. It took me a couple of years of just maintaining that. Thankfully, I had autonomy, so I just did it by myself, slowly changing a thing here or there without major interruption from bosses or anything. Then I realized what good code actually looks like

and that what I had inherited was actually just clever, but terrible. And oftentimes clever is terrible. And there's some clever stuff in this code base you're talking about. Yeah. Things that you're like, Oh, that's kind of clever,

but it's also so dumb. I think that was one of the, the lessons I had to learn was just like how clever you can be and how much you can solve a problem with like the most complicated code you could possibly imagine. And yet, for the end person, they don't care. It doesn't really matter. But for me, I don't know. I've never been a business-y person.

I like coding because I think coding itself is really interesting. And so for this first job, it was kind of a shock of like, this is what I have to deal with? Like this ugly grossness? But I think that that's changed a bit. I look back on those moments quite fondly. So yeah, maybe I should talk a little bit more in detail about, I'm assuming most people haven't read the blog post or whatever.

Happy to fill in some background of like, what was this code base look like? What are the weird things going on in it? Yeah, absolutely. I think we can drill in on specific aspects and just enjoy them as we do, but you can definitely paint with broad strokes in terms of, I already said it's a big visual basic C-sharp thing. The database was massive. It had lots of columns. You go ahead and fill out some of the big picture aspects of what this thing was, and then we can dive into some of the details because there's a lot of just enjoyable

tomfoolery going on. Yeah, so let me paint a little picture of the company. So this is a big, like I said, a big credit card processing company, but the actual office I'm at is almost all staffed by customer support people there's just this like one room that's developers probably like 80 developers or so in this one big huge open office and they're all we're all writing this software for these customer support people it's like totally separate off from like the rest of the credit card processing business. This is all bespoke software built up over the last 10 years.

This is like 2012 is when I joined this company. So it was a little bit behind the times, even for then. Very stuck in the past of how software is being done. But I don't know, it's your typical, the customer support people are all really serious, have to dress a little nicer, and the developers are all kind of chilling, shooting each other with Nerf guns

and being a little bit more wild. But the code was the kind of part that everyone wanted to ignore. The stuff I was on was, I started as an intern. It was like the legacy code base that the professional developers didn't want to touch anymore. There were all these teams who were going and redoing the big rewrites. And the interns and a couple developers were kind of shoved on this old legacy project that was just

massive. So the database itself, I talk about in the articles about we ran out of columns. The database is this massive database with the merchants table, which has 1,024 columns, because that's the most you could have in SQL Server.

The code base is hundreds of thousands of lines of C sharp and VB. And the reason it's kind of split is they decided halfway through to change from of lines of C sharp and VB. And the reason it's kind of split is they decided halfway through to change from VB to C sharp. But like the whole way it worked was like session state got stored in the database

every single time you changed pages. So it could like swap back and forth between the like VB and C sharp world. It's this like crazy bespoke IIS setup that takes like days to get running on your machine. The whole thing was just, yeah, kind of duct-taped, terrible code base, every JavaScript framework you could possibly imagine.

And the task as an intern was, here's a big list of bugs and features that we don't want to actually spend developer time on, this is your job now. You got to ask yourself, how does a code base get to this point? Like, is it because there was no leader? Is it because no one cared? Is it because it was sort of siloed off the seemingly primary application,

which ran the processing? This was, as you said, customer support. Maybe it's a sidecar to the business and less important, but you talk about sales folks putting their wins in there and logging in because you've got the calendar and stuff like all these, like, how does that even happen? Is it because there's no leadership? What do you think? Yeah, that's a good question. So like in some ways it's, you know, less yeah uh it's a good question so like in some ways it's you know less important because it's customer support but it's also yes the sales organization is kind of in this code base right like that that really matters but also shipping out payment terminals all happened from

this site or at least a lot of them happened from this site they might have had other locations but like so shipping was also a big part of this So it's not like it was like completely to the side based on my time there, obviously. And like all the stories I heard, it's really leadership conflict, constant changes in leadership, constant people coming in and out at the top level, different directions changing. One of the things that was a really kind of annoying aspect of these codebases was the names for every product had changed a hundred times. The names for every like, there's like this big sales hierarchy

and it really, really mattered what the sales hierarchy was. Like there's the regional manager and the district manager and the, you know, all these different terms. And they were just abbreviations in the codebase. It'd be like the DM, the RM, manager and the district manager and all these different terms. And they were just abbreviations in the code base. It'd be like the DM, the RM. But those abbreviations had changed over time to mean the same things. It would be like district manager versus direct

manager. And so you'd actually have to look to know what those entities refer to. You'd have to look at source control and then go talk to Munch, who was as I put the resident shaman, and ask him, what things in this year did this refer to? So it was just this constant change, right? And constant change in direction, constant change in leadership

that just made the code base go in so many different directions as well. I think, you know, Conway's Law ends up winning out on all code-based organizations. You knew Munch, obviously, but if you didn't know Munch, would you think he's like a Dwight? This reminds me of The Office in a way. Instead of paper, it's a software package or program.

Just getting the job done, basically, is the mission. I can totally see that. I don't think I would call munch a dwight munch was a very charismatic person he was somebody who has management wanted him to be in charge so many times and he just refused to to go be promoted beyond that so he was like the de facto leader of so many things, but his job title never reflected it. He was, I think his job title was just like systems analyst or something like that. He wasn't even technically a programmer by,

you know, the org chart, but he, he did so much. So he, I could definitely see, you know, the,

the Dwight comparison, but I think that'd be a selling him a little short. He was a, he was just a really kind hearted, really nice person. At any time you needed help on anything, he was the one who was willing to sit with you and explain it. He was a great storyteller. I mean, I think shaman is the best descriptor I can give for Munch. Was his name actually Munch? I mean, who names their kid Munch? Okay, so his name was not actually Munch? I mean, who names their kid Munch? Okay. So his name was not actually Munch. Apparently there are only two people in the world that know why his nickname is Munch.

These were two high school buddies. And he tried to get rid of the nickname when he went to college. He tried to just go by his normal given name. But then there was somebody else in his dorm with the exact same name. and one of his friends knew him as munch and they just said oh we can just call him munch it's fine and then a second move he moved to like a new town after college and tried again and yet like some he ended up running into an old buddy who called him munch and then it like spread and so even his wife apparently didn't know how he got the nickname Munch. That is hilarious. Some names just find you.

You know, you just can't, you can't escape them. They just stick. Yeah, I remember the day where we got a new system for being able to, like, naming. Like, we got a new, like, AD setup or whatever so that, you know, kept your emails and your names or whatever. And it turns out that it had a little lower permissions requirements than the last one. And Munch went and changed his name and everything to Munch. So instead of actually having to be like, oh yeah, that guy, that's Munch. Now in the system, it was properly Munch, which was fun.

He finally embraced it, huh? Yep. Nice. So let's go back to these columns okay yes okay so there was a merchant's table it hit the max number of columns 1024 at the time apparently was it sql server has got more memory now you can now add 4096 columns so yeah yeah good for them but because this arbitrary i mean it's arbitrary, memory constraint forced them to either stop making new columns or create a second table in which you can just shove some more

columns. That was the choice that was decided. So now there's merchants and merchants too. And I assume every time you look up a merchant, you just got to pull both tables in, don't you? Yeah. I mean, it depends on what columns you're grabbing right you have to know what things are are where and most applications i mean most of these columns were completely pointless i am sure if you actually looked at them they were completely not used for eight years or something right that's just like somebody decided i'm just gonna add another column all of our like sql schema was not actually in any code any source control they you just had to submit forms to the dbas who would manually run them for actual like stored procedures of which there were

hundreds or thousands you had to like manually check them out in different environments and then write your name like i am editing this please don't edit it uh and then you would like leave a little like change log of what you did and you'd end up finding out like someone didn't do it in the lower environment so by the time you get to production it's wrong and all sorts of uh fun things there so you know adding a column as long as you can get one dpa to agree you're good to go. Which they did thousands of times, apparently. I just wonder, how can you possibly have that much information about a single entity in the world? Obviously, there's probably a lot of foreign key relationships, which is probably a lot of those columns. But I mean, how much can

you know about a single merchant? Like more than 1,024 things matter? Yeah, it was, I think, you know, a lot of it were, was things like, are they involved in this promo? Do they have this kind of equipment? You know, there was all sorts of different iterations of this product or like, you know, who is their salesperson? So there, there was honestly, like the domain itself was very complicated. I never got a full picture of all of this domain of what's going on. Now, as we now know, of course, and even then it was happening, like Stripe has simplified payments to like a nice minimal thing. And I actually remember at the time, there was a group that was supposed to be doing like open source work

and trying to take on Stripe. And I was so excited. I was gung-ho, 20-year-old, first job. I was like, I'm going to join the cool group doing open source. And they had some Python code that looked straight up like C-sharp code. And I made a PR to fix their indention,

and they got really mad at me. They were using tabs instead of spaces. You know, it's Python, like four spaces. And yeah, no, it did not go well. But yeah, like I think, you know, for me, a lot of this was just like, I didn't know anything about like the broader tech ecosystem. Like I never even realized that I could do this as a job.

Like growing up, I just even realized that I could do this as a job. Like growing up, I just kind of learned programming on a whim and had no idea that people, the stuff I was doing was what people actually did. Really? Yeah. Yeah. Uh, the, the way I got into programming was a bit weird. Uh, I grew up relatively poor, you know, not always having food on the table poor but you know by i'm still have a roof over my head still electricity you know the u.s poor right obviously there's people who have it much worse in the world but we did have a computer luckily we were gifted a computer from some family friends but my brother would always hog it my older brother and so one day he decided that uh i got my own computer because he found one

by the trash with mud on it it was this old dell with windows me on it and he just gave it to me and said this is your computer now this is the one you have to use it didn't work i didn't know the password for windows me but luckily i had a friend who was uh whose dad was a system admin and he had mentioned lin to me one time. And so I got Ubuntu running on it. And I was 12, had no idea what I was doing, just burning CDs, trying to get some software to work on it.

I ended up getting Ubuntu on it, ended up getting wireless card drivers working for it. Wow, that's an accomplishment. Yeah, Indus Wra wrapper was this like taking windows why it didn't have a wireless card i thought okay i can buy this and install it and it'll just work but indis wrapper was this way of like wrapping windows wireless drivers and trying to make them work for linux and i now know what i did was i compiled from source indis wrapper by burning cds to get the dependency tree.

It was the only way I had to transfer data from the computer to between computers. And so I like burned these CDs and compiled from source in Diswrapper and I got it working. And that was when I was just hooked. Like this was so exciting that I got my own computer working. By doing a bunch of magical incantations, didn't understand at all that's amazing so you would download them on your brother's machine yeah and you would rip them to a disc to a cd which is a one well maybe one time but yeah you pretty much have one shot at it right oh yeah i didn't have readable writable i luckily had gotten some some yeah right once and you sneaker net that

sucker over to your machine yep and then are these just tire balls or how did you do you remember yeah yeah i when i when my brother was gone to a friend's house this is when i would go and you know sneak on and go do this and get things working uh so yeah i mean he was he was nice if i really asked him to let me have computer time it's like he completely kicked me off. But you know, your older brother, right? Like when you're that age, it's like, ah, I can't bother him too much. So yeah, I got it working.

And that was my computer for probably five years. It had like 128 megabytes of RAM. It was, you know, at that time, it was quite a bad machine. But with Linux, it ran super smooth. It was great. Linux is the best. So this was your first gig, really, this place. And you said that you were self-taught as a programmer.

So how much experience did you have before this job doing programming to know that this was bad? Like, not the way yeah i had only really i mean i'd written a lot of my a lot of code in my spare time you know i did a bunch of like project oiler.net i don't know if you all are familiar with it but it's like for anyone who's not listeners it's a bunch of math problems that you do have to use programming to solve so it's a bunch of like number theory kind of stuff i'd solved they're like they get really difficult really fast like i think i've solved all of them i will ever be able to solve in my lifetime because like as they

continue on it's just like graduate level math and so i had done a bunch of those i had played around with like uh mozilla had had a new extension method called Jetpack that they were playing with. I'd made some of those and released some and people used them out in the wild. But they were all like super small programs. I'd written a program for my school. I had a great programming teacher. Miss Johns was her name. She was fantastic. I love her. She did not know programming but she was a great like great person who would teach from a book and recognize that like i knew more programming than her so i just was a tutor in the programming classes like i i technically i took the java ap exam never took the class got a five on it like i i did a bunch of stuff in my spare time so like i learned java i learned python

i just i just loved programming. It was my main hobby. I'd skip my calculus class to go program. That was the kind of... Failed a lot of my classes because I would just skip them to go programming. Well, it paid off. Yeah, Ms. Johns was actually the reason I got this job.

A few years after high school, she had reached out to me and said, hey, there's this company, uh, looking for interns. And, uh, she told them about, um, a story that I I'm happy to tell about a time that the secret service busted in my door for hacking. And they were impressed enough to give me an interview despite, you know, not going to school. So, um, so so yeah happy to tell that story but i also you know no we're here to talk about the code base so no uh secret service knocking down the door i think is a worthwhile diversion don't you think adam yeah is you got more to the story what's the story why

i mean were you i mean you you shallowed it deep it uh okay so yeah so for people who don't know like the secret service also deals with like internet security. You think of the Secret Service as just being the president, which is what I thought when they busted in my door. I worked at a local or like regional grocery store as a cart pusher when I was in high school. And the system to like check your schedule was the most convoluted, annoying system you could

possibly imagine. You had to get on a VPN, then you had to log in like four times, and then you could finally like traverse these links to go check it. And they would publish the schedule Saturday night for like the Sunday morning. And I just got so annoyed with having to like spend like 15 minutes logging into this system that I was like, I'm going to write a program. It's going to do all this for me. And it's just going to email me my schedule every week.

So I go to go to write the program and I start, you know, it's a simple little Python program making some HTTP requests. And I was like, all right, let me figure out if I don't send a password, like what error I get. And then I can, you know, continue on from there. I got the schedule. I didn't send in a password, username or password and just got the schedule. And I was like, that's a little weird. Did I somehow like, I don't have cookies set. There's no way that it knows my credentials. So I visited in the browser and I was like, oh, actually that inner iframe there doesn't require auth at all.

And it's a schedule though, is what I thought. It doesn't really matter. It's just like someone's schedule. So I pushed the back button on the browser, or I clicked the back button in the app, and it takes me back to another page with my social security number and my bank account information.

And I realize every single employee's social security number and bank account information. And I realize every single employee's social security number and bank account information is just on the web, unauthenticated. And all you need is this employee ID, a sequential number to find them. So I try to reach out to the company to tell them, hey, this is a problem. Because I knew my local branch is not going to know anything about this. I try to reach out to the company to tell them, hey, this is a problem. Because I knew my local branch is not going to know anything about this. I try to reach out to corporate. I filled out an employee survey and stuff.

And they respond back to me. This is a customer support person. My older brother told me, well, you can't give it to them for free. You can't tell them the security vulnerability for free. They should pay you for this. Me being the dumb 16 year old I was. Dubious advice. I was like, well, you know, you'll have to pay me. Didn't hear back for several months. The plot thickens. All of a sudden at 6am, I hear a bang

on my door. I sleep like on the second story, but like my room kind of goes straight down the stairs to the door. So I'm like the closest to the door of my family. And I hear this like really loud bang on the door at 6 AM. I go downstairs and I hear police open the door. Like it was like that, like pitch. And I look outside and there's no police cars, nothing,

not a single one that I can see. Cause like we have a big window and I look out, there's nothing out there. And like there had been a break in and a house not too far from us like a week prior. And I'm just like, this does not sit right to me. I was like, how do I know it's the police? And the guy responded very confused and he goes, um, well, we're busting in the door. I back up battering Ram into my door immediately throws me on the ground puts me in handcuffs and in walk secret service agents the wildest thing i have a i i don't know if i sure still have the picture but they like left a big huge mark in the front door they had the

battering ram ready they didn't even wait very long. They just came busting in. Came busting in. And so I asked the first Secret Service agent that is coming in, can I know what this is about? As I'm on the ground in handcuffs and he goes, no, that's classified. Next Secret Service agent, which I'll just call B. I'm not going to give his name even though I know it. Secret Service agent B goes, what? No, of course he can see the warrant. Hands me the warrant, which shows the grocery store name. And I know, you know, okay, that's what this is about.

I tell the whole story to the Secret Service Agent about what happened, what I found. And I kid you not, he goes, um, well, I think someone overreacted here. But you're facing pending charges of 30 years in two states and at the federal level. Wow. I think my favorite moment from that, though, one. So I had this, you know, computer that I that I had gotten.

I still was using it at the time and I like changed out all the parts in it. And I had this like backpack full of old hard drives they like tore up my bed like flipped over the whole room like it was absolutely insane there were eight local police and two secret service agents and they don't take the backpack full of hard drives that was sitting right next to the computer which is just like top-notch police work yeah and then the second was we had gotten an iMac at that point and I watched as two local police looked around and they're like where's the rest of it and one had to another guy to be like I think this is the whole thing that's all it's just all in one it was great um so yeah I I didn't do any real hacking. I just found something unauthenticated. It was not complicated.

I was not some massive hacker. I ended up getting interrogated by the company for like eight hours later. Did they just dismiss the charges? Or how did it turn out? Yeah, no charges were ever filed. It was just a search warrant. It was pending charges based on what they found,

which, of course, there was nothing. I didn't do anything. I just found a security vulnerability. But my mom made me tell the company because she was worried I wouldn't have a job or whatever. And I knew they didn't know I worked for them, which was true. They were a very disorganized company. But I ended up getting interrogated by some lady from corporate who, when I asked for a lawyer to sign documents, she stood up and screamed that I threatened her and i got fired not for anything i did there but for threatening her wow yeah so that story plus miss johns

together got you this job because they're like well you must be good what he does secret services after him yeah yeah uh i don't think she even knew the you know like technical details or whatever but it was enough they were intrigued that they let me get an interview and luckily yeah yeah i got the interview i think six weeks into my internship i got hired on full time and yeah were you actually good i mean you know i could fake modesty but yeah i was good i mean i wasn't good compared to now. I was awful. If you look back at all of that stuff, I was terrible.

But for where I was at as a junior developer, basically like a recent grad, I knew my stuff. When I joined, we took the backlog from like 60 items in the queue that had been there. And I was able to get like 40 of them solved pretty quickly. I mean, it's why I got hired in six weeks

from intern to junior developer. Turns out like all the stuff I had been doing in my spare time was actually real software. I just didn't realize it. That's cool. Yeah. He's the kind of kid who has got a backpack full of hard drives. Of course he was good at what he did. You don't just accidentally end up with a backpack full of hard drives.

I really didn't. And I think I'm sure like part of the reason I want to share this is not like to brag about myself. It's like I, because I grew up, you know, like none of my family had gone to college. It was a very working class family. I had no concept that this was something. I read a bunch of tech articles. I'd seen all of this stuff. And I knew some people out in California did this. But as a kid growing up in a small town in Indiana,

I just thought what I was doing couldn't possibly be the real thing. And so I was two years out of school. I'd never once considered like programming as a job because like, I just didn't think I was good enough to do it. Right. Then you find this code base and you realize how many hundreds of thousands and millions of dollars in labor have gone into this monstrosity. Exactly. Right. And I've realized like, ah, you know, my worst code is, you know, I write bad code.

I wrote tons of bad code at that company. Zero question. I mean, they had to scrap a whole project that I did. Like I made all sorts of mistakes, but like you can not be perfect and yet contribute quite a bit. And these people, yeah,

they created value for the company despite this code being awful. Okay, friends, here are the top 10 launches from Supabase's launch week number 12. Read all the details about this launch at Sup superbase.com slash launch week. Okay, here we go. Number 10, Snaplet is now open source. The company Snaplet is shutting down, but their source code is open. They're releasing three tools under the MIT license for copying data, seeding databases, and taking database snapshots. Number nine, you can use pgreplicate to copy data, full table copies, and CDC from Postgres to any other data system. Today, it supports BigQuery, DuckDB, and MotherDuck with more syncs to be added in the future. Number eight, Vect2PG, a new CLI utility for migrating data for vector databases to SuperBase or any Postgres instance with PG vector. You could use it today with Pinecone and QDrant. More will be added in

the future. Number seven, the official SuperBase extension for VS Code and GitHub Copilot is here. And it's here to make your development with SuperBase and VS Code even more delightful. Number six, official Python support is here. As Superbase has grown, the AI and ML community have just blown up Superbase, and many of these folks are Pythonistas. So Python support expands. Number five, they released LogDrain

so you can export logs generated by your Superbase products to external destinations like Datadog or custom endpoints. Number four, authorization for real-time broadcast and presence is now public beta. You can now convert a real-time channel into an authorized channel using RLS policies in two steps. Number three, bring your own Auth0, Cognito, or Firebase. This is actually a few different announcements. Support for third-party auth providers, phone-based multi-factor authentication, that's SMS and WhatsApp, and new auth hooks for SMS and email.

Number two, build Postgres wrappers with Wasm. They released support for Wasm, WebAssembly, Foreign Data Wrappers with Wasm. They released support for Wasm WebAssembly Foreign Data Wrapper. With this feature, anyone can create an FDW and share it with the Superbase community. You can build Postgres interfaces to anything on the internet. And number one, Postgres.new. Yes, Postgres.new is an in-browser Postgres with an AI interface. With Postgres.new, you can instantly spin up an unlimited number of Postgres databases that run directly in your browser and soon deploy them to S3. Okay, one more thing.

There is now an entire book written about Supabase. David Lorenz spent a year working on this book, and it's awesome. Level up your Supabbase skills and support David and purchase the book. Links are in the show notes. That's it. Superbase launch week number 12 was massive. So much to cover.

I hope you enjoyed it. Go to superbase.com slash launch week to get all the details on this launch, or go to superbase. com slash change law pod for one month of super base pro for free that's s-u-p-a-b-a-s-e dot com slash change law pod you didn't introduce the calendar did you no yeah so this is uh there was a calendar table which was literally a hand filled in calendar that my understanding at best was that when contractors logged in versus when employees logged in, there were certain days that employees were allowed to log in,

but contractors on weekends and holidays could not log in. It was supposed to be completely forbidden. And this was the system that was doing it. And so one time, they filled it in for a few years, and it actually ran out. And they had to scramble to figure out how to log in, and then just had an intern fill in another five years had no idea which there were so many services so many programs running in the background so many cron jobs they had no idea which one had been

locking people out i love it so just filled it in more just one row per day and you just got to fill it out and it's going to check against that row for the day. And if there's no row, then they can't log in. Exactly. I also had a task that I don't mention in the article where there was this like 5,000 line Pascal program that had been apparently like very mission critical for the last eight years. And all of a sudden had started failing.

And they had no idea why. And I was told to rewrite it in C Sharp because they didn't want to support Pascal anymore. I looked at this thing. It was just, it was only 5,000 lines because it was copied and pasted. Like they unrolled a loop by just like copying and pasting code over and over again.

So my end code was like 200 lines. And I get it. I make sure that like everything I'm seeing is identical. I was really thorough to make sure I got the program right because apparently it was really critical. I was given a week deadline. Okay, I get it. We put it in service.

And what it was doing was sending a bunch of emails about some process or other. I didn't know the details, but reading from a table, sending emails based on reading from the table. Not complicated. And I immediately, like the day I turn it on, my manager comes over to me and says, please turn that program off.

We are getting constant emails from people. Apparently, this program hadn't run in eight years. It was about something that was eight years old and it was spamming people's emails every half hour Apparently, this program hadn't run in eight years. It was about something that was eight years old, and it was spamming people's emails every half hour because it couldn't find the data it was expecting. So it was defunct, basically. It had been not running. It had been running incorrectly for eight years.

They just finally turned on alerting, and that's why they started noticing that it was failing. Oh, okay yeah so you did implement the correct behavior i've implemented it exactly right it's just that uh yeah nobody knew that it was failing because that program that you know part of the business had been gone for eight years but those people were still there uh so, yeah. Who knows, man. Yep. Well, that's wild.

Yeah, I think the other one that I loved was like, there were whole programs in source control that were just decompiled sources. So they were C-sharp, but they had lost the source code to them. So they just took the binary, ran them through a decompiler and checked them into source control okay and you had to make changes to this application

and i don't know if you've ever worked with like c-sharp decompilation but it is imagine it's unreadable right it is completely completely unreadable. It is compiler output. It's like if you had to work on JavaScript minified code. And I had a person, one of these was our time tracking system. The way we tracked all employees was a custom built application, but we had lost the source control, lost the source code. So it was decompiled. And there was some list that was wrong, according to a business person.

So I go in there and I'm looking at this like boolean logic that's been like optimized by a compiler and i i finally like get it i write down like the logical formula and i plug it into wolfram alpha and it it tells me like the simplified form oh cool it was great i was so proud of myself for thinking of doing that because it was really complicated it turns out to be like this and that or this and that or this and that. That was like the whole entire thing. And I figured out what each of these variables came up with. I was so proud of myself. I go to the business person. I'm like, all right, here's all the variables. Here's the Boolean logic. What does it need to be instead? And she's like, oh, I don't know. I don't know what any of those't know. I don't know what any of those

variables mean. I don't know what any of these terms mean. Instead, I had to go change a variable, print out a new list on a piece of paper, bring it to her desk, and she would mark which ones were right or wrong. Oh, wow. Best in check. And then I would try to figure out based on her marks what the bullet... And we did it. It know, 10 rounds of printing out pieces of paper and letting her mark on them. But yeah. And then did you decompile that and throw it in source control? Or like, how did you recover from the circumstances was no, you just now edited that that was the source code, right? That's the only thing we have is this crazy decompiled thing. I cleaned up that little bit there, right? Made it a little cleaner and added some human readable terms to it, but I mean, there's no way you can fix it. It was probably a 30, 40,000 line application,

right? No way you're going to rewrite all of that in the time given. Why did you stay here? Why did you keep doing this job? Was it just like a weird conundrum slash challenge? Like this cannot be real and I must stay to see what happens? I mean, you got to realize this. I mean, I didn't stay there super long, but that was, I probably would have, but I met my now wife and moved.

But like, I mean, there were tons of people who'd been there five, six, eight years. This was, one, a small town with a big city near it, but there's not a ton of programming jobs. And all of them are kind of equally crazy. I knew people who had worked at other companies now and come here. And I think this kind of stuff exists a lot more. I mean, there's tons of comments, right? Being like, I thought, just like we had here,

it's like, I thought I worked on this code base, right? Like, this sounds very familiar. But also, I didn't know any better. I just assumed, like, this is what code looks like, actually, in the real world. You know, I'd seen some open source stuff, but never really dived into it. But I'm like, oh, maybe open source is better.

But at a company, this is just what you have to deal with. And while it's definitely the most story-worthy code base I've worked in, all of the things that were bad were just so obviously bad, it was not a bad place to work. I would actually rank it on one of the better places I've worked. Not the best, but one of the better places I've worked. Part of that was definitely my position in there. I had a great manager. He was just so supportive, so nice, always made sure that I got the growth opportunities that i needed to like become

a better developer he saw like the potential that i could do and made sure to like help me and get people you know get more senior developers to help me learn stuff and challenge me that was really good uh but also like i mean this will show my very strong bias here uh that i know you all might not necessarily agree with, but there was no like agile process or none of that stuff, which I have found to be like the main factor in job dissatisfaction for myself. So like the lack of process was so freeing and so nice. Yeah.

Well, you're not going to get a disagreement from me on that one, but maybe we don't do agile around here. We do know whatever we want, basically. We just code stuff. Yeah, I wasn't sure. I don't know exactly, but with the Kaizen and all of that, obviously. Well, the Kaizen just means

continuous improvement. That's something that I'm sure you're into, right? Let's make things better all the time. I mean, your Kaizen episodes are great. Just wasn't sure. I know it's always contentious when I say not a fan of Agile, even with a lowercase a, just not a fan. It was a good company to work for.

The biggest problem, really, why I wouldn't have stayed there longer is being underpaid. You're in a small town. There's limited talent. There's limited places that you can go, though, so they can pay you way less. Turns out I was making the same as people who had been there eight years. And who had job titles way over me, which was just wild.

But this is programming in the Midwest. I know tons of people who are at companies like this with equally crazy code. I mean, even Salesforce has a branch here in Indianapolis. And all the code I hear how the Salesforce sounds similarly wild to this. When you have so many people and so much, I guess, momentum, you have to make progress. And sometimes progress is like, just duct tape that part. And that's kind of what the employees table, this seems like duct tape. Why in the world do you have to drop this table at 7.15 every morning

and then repopulate it with a new injection and then people can't log in? Why is that the way? Or the same thing with the sales numbers. Why do they have to claim these wins and then put it on this board and they were able to subject these interns to doing all this minion work basically to get seemingly their numbers projected properly. I don't know. It's hard to decipher exactly what's happening there. Yeah, it's a little complicated.

The basic idea for that one, just to be clear, is like salespeople, obviously there's the real accounting. I know there were some comments on Hacker News of like, that just sounds like fraud. There's the real accounting system and then there was the rewards system where they get bonuses and stuff. This was the reward system. And, you know, yeah, they were trying to get bonuses every month. And if they made a big sale at the end of the month and they had already gotten their

max bonus, they would just move that to the next month so they could get their bonus for next month. Right. So they had a cap on commissions and things like that and this was moving it around that's funny because i was encouraged when i was in sales back in the day like hey let's just move that sale to next month because you've already reached your quota this month like good good for you yeah and it got encouraged here but the way that it was implemented was interns manually writing SQL statements. Take that hourly wage out of your bonus.

It costs a little extra. There's some intangible cost there on that bonus system. There were people who the whole internship they were there, they never got to write any code other than these SQL updates. I think that's a shame. That's not how you should treat your interns. But that was one of the things, I just refused to do it. I just never wrote one. So, you know,

every time it would be assigned to me, I would just go find something else to work on and do that instead. I wasn't a great employee also, to be clear. I was 20. I was very, you know, I was very a little very, uh, a little bit more prideful than I should have been a little bit more arrogant, uh, than I should have been for sure. But yeah. No. What about these hard drives and gilfoil? Is this a Silicon Valley nod or is this,

uh, this is a real gilfoil? It can't be a really a gilfoil, right? No, no. It was definitely a Silicon Valley reference. Okay, good. Yes reference okay good yes yes uh i i did not do it as bait for this show but i i guess inside i should have uh thought of that as a benefit too yeah it definitely was like when i saw that i thought adam's gonna love this there's like there's someone named gilfoyle you say let's call him gilfoyle so i figured and then i immediately command f'd and typed in sil and found-L and found nothing. Or Silicon Valley. I thought maybe at least reference it. Let's, you know. No, I figured it for anyone who knows, you know, is a good reference.

And anyone who doesn't, it's a good enough name, right? It's kind of a fitting name, I feel like. Sure. So yeah, no, his actual name. I never, I wouldn't, I felt a little awkward like using his actual name because I never met him. I have no idea who this guy was other than through his code and so yeah he was he now i don't think i looked him up i don't think he's a programmer anymore he uh sells exotic pets so yeah uh i thought gilfoyle just

seemed like a fitting name especially if you know the reference, right? That was always the sense that I got of this guy. And yeah, he was, I mean, the most prolific programmer I have ever seen. The sheer amount of code in that code base, that was his. And the sheer amount of applications that random customers or customer support people would have that were just from scratch Windows applications that he wrote with complicated logic. Anytime, because he refused

to use source control, which was why we had his hard drives raided on Munch's desk, he refused to use source control. If a person asked for a code change, a feature change, he would just rewrite the application from scratch. And he was apparently, from everything I heard, that fast that pumping out a brand new 10,000 line application in a day was not unheard of for him. But the code... This is like a legend.

This is like a legend. Yeah, total legend. Yeah, absolutely. And because he didn't use source control, I have no idea how fast he actually wrote this code, right? Like, so you don't get any history on it, but like. That's how the legend continues, you know, he can't have a trail, no paper trails.

Yeah, I wrote this in a day. Maybe he just anticipates needs way in advance, right? Or like puts his code through some obfuscation every single time. So it looks like it's slightly different right uh i don't know what it was but yeah his code was it was a trip to try to understand though there was just every time there was never a consistent pattern to the craziness it was like he just woke up every day and, what new weird programming pattern can I abuse here to write this application?

Services that were pure functions that I literally do not understand why they existed. Clients that were just like these super thick clients that, like I mentioned in the article, completely empty classes. Classes, I did not exaggerate in the article. They were empty classes. They had a class definition, method definitions, but there was no code in any of the methods. And it was all to build up a structure and the hierarchy would go 10 layers deep of inheritance. And it was all to build up the structure that then would become a pipe delimited string sent over a socket. What? the structure that then would become a pipe delimited string sent over a socket what that was all driven by the database and so like when you looked at it you were like okay what does even happen like once you even figured out like oh this is a data structure not a class hierarchy it was like what do they turn into oh well it's like dynamically choosing which column of which table to go grab this field from and now it it's custom there. And then the table would encode,

like how do you encode the value? So like you could infinitely configure how the delimited string would be, would be created. But like, there was no reason to do that because it's an old application that hasn't changed in 15 years. And the same message was written every time it was, it was wild stuff but i actually

debugged that bug for a year it manifested itself as like 15 different cases in the system where things would just go wrong and i thought it was like a memory leak for the longest time i thought it was all these things and like finding out that it was just some legacy third-party application reused unique IDs every month was the most exciting and most let-down bug find I've ever seen in my life. Yeah, not exotic. I thought for sure it had to be Guilfoyle's fault, right? His code was too clever.

I really wanted to blame him on his cleverness. I was going to say, it's really convenient sometimes to have a Gilfoyle. It's like a patsy. When something's wrong, you've got someone to place the blame on because he's been prolific and he's done all these things and he's not around. So surely, gosh, Gilfoyle.

What's wrong with you? But no, this was a third-party system. What about ops in this case? You're talking about the code base, but somebody's got to keep that database up and it seems like it's getting hit in like wild ways like this chain function for example it's probably got the database spinning the disks are spinning and i'm assuming that's the day of hard drives yeah that so the the database was definitely we had quite a few DBAs. Like the DBA to programmer ratio was pretty high. And we had some very beefy machines running this SQL server setup. On-prem?

On-prem. Obviously. So yeah, we had everything. We had a, like actually at the office I was, there was like a data center area as well. It's like a shipping area. So it was on-prem, it was local. There was some talk about the company building a private cloud system and things like that, but it's a little early for them to actually do that. Nothing ever came of it. Yeah, there was definitely some beefy things. Most of ops though, really ran through one guy who was

really good at his job. He was the ops guy. I mean, there were technically other ones, but everything I ever dealt with, it was him doing it and maybe delegating some tasks occasionally to other people. But it was a pretty bespoke kind of setup. Deployments were all done by hand by him late at night, so that way it wouldn't affect anybody. It was that kind of place, that kind of setup where servers had pet names and all of that. It was well before. There was some early maybe looking at Puppet,

maybe doing some of that, but nothing really came of it. So yeah, it was a pretty... This is honestly one of the things that was so strange is the scale of the actual code base, the scale of how many people are using this is small but not completely trivial. And yet like this, this legacy code base,

especially the other ones, the other applications that were like the big rewrites had not seen production use. And we were able to, it was like me, a analyst, a QA, one senior developer,

and like four interns were able to out compete like all of this rewrite where like they were we were adding new features fixing old bugs and doing all of that in this small in this system while they were off doing their big agile processes and doing all their story pointing and all of that and never getting anything done so it it was fun. When you actually think about it, if you're not one of these big web scale companies, servers are simple. Code's not that complicated, even if the code's complicated.

You can make changes if you just don't get in your own way. So yeah. It sounds fun. I mean, I would like to work on this code base for maybe like a month and then move on. But I'd like to visit. As a game, maybe. Yeah, it yeah it is a game and i mean it's got to feel like that to a certain degree it did it felt very much like a game but i think that's how i approach most of this work like like i said like i think the job i'm at now we've got a we've got a big massive

old code base i i now work on a fork of Rhino.js. Okay. I don't know if you all remember Rhino. I do, but I can't remember what it does. But I remember Rhino.js. It was involved in my cappuccino days. I think they were using it back with Objective-J. Yep, yep, that sounds about right.

It is a JavaScript implementation written in Java. It's a compile to JVM bytecode and an interpreter, all written in Java. It was abandoned years and years ago, but I now work at a company that has a long-term fork of it that runs millions and millions of lines of customer JavaScript. Really? And it's got some custom features. Say more? Sounds good. Yeah, so we're now trying to refactor away from it.

But for example, in our original version of JavaScript that people still use to this day, if you did dots, it was like doing question mark dot. So if it was undefined, it wouldn't throw an error. It would just return undefined. This was a choice by the founder of the company. I guess I just have a knack for finding these code bases. I don't know what it is.

Where things are just crazy. And I think a lot of developers work in these kinds of things, right? They just don't talk about them publicly. Totally, totally. I've had similar experiences, all I think at smaller scales, both in terms of company size and code base. My craziest one was I inherited, I did a rescue project for a boat shop somewhere in Georgia

where they just needed, basically it's a Ruby on Rails application that ran back office for a boat shop. A lot of merchants, a lot of sales, a lot of this kind of stuff. So similar tables and stuff, which is why it's probably resonating with me. And they had lost the original dev team.

It was like a contract team came in, built the system and left. And then the IT guys who are also third party the original dev team, it was like a contract team came in, built the system, and left. And then the IT guys, who were also third party, kind of took the system over because they were just nice guys who were helping out the company. That was a successful boat retail shop and relied upon this application to run their business now. And this was like really early Ruby on Rails days. I think it was like version 1.2 or something.

And so there's a lot missing. And this team came in and wrote a lot of very clever code, basically implemented a meta framework on top of it. And it was insanity. It took me a very long time to unravel. And then they left, you know, and these people were left kind of high and dry. And so I was happy to help out. And I had the challenge, the game, like all these things. There was no development system. It was like production and I copied the code down and try to get it running on my machine, you know? So you're very much like doing crazy stuff,

very small increments at a time, trying not to break things. And so that experience just resonates with everything you're saying, like the metaprogramming, specifically when you're talking about the thing that generates data structures from classes and the database kind of is the programming system as well, if you want it to be, but no one's using it.

Like all that stuff was there. And it took me a very long time to be able to unravel it and understand it to the point where I was like, oh, it's kind of clever once you know how it works. But that's a terrible thing. And so I think, and I didn't write up about that at all, I think there's tons of codebases like this out there. Yeah, I wish I had finished this write-up. I haven't actually done much Ruby,

although I did work at Shopify on YJIT, the Ruby JIT compiler, a little bit before getting laid off, sadly. It happens. But the one Ruby code base I worked in, there was just this little bit of super clever metaprogramming, which, of course, Ruby loves.

Lots of people in Ruby love their metaprogramming. And I could never find, the reason I never wrote it up is I could never find a non-circular starting point. Every time I would want to explain something, I would have to, like, it would, because the code, like, meta-looped on itself every single time.

There you go. See, that's the problem. Yep. It's like a time travel movie, which we talked about pre-show, so not a good callback. But yeah, yeah, yeah. That's one of the problems with meta-programming

is it's just too meta sometimes, and you can't unravel it. Yeah, I mean, I think this is, not to self-promote or whatever, but this is one of the reasons that I really enjoy the Future of Coding podcast. Because I think there's a lot of code out there that people aren't happy with and I don't think we talk about it.

In some ways, I think looking at this legacy code base is the same thing for me as looking at the shiny new stuff. I think oftentimes we don't appreciate code for what it is. We don't look at what's come before. We kind of look at legacy systems like they're all bad and there's nothing good to gain from them. I've seen a bunch of blog posts talking about crazy code, but one of the things I wanted to do in this article

was talk about crazy code, but in an endearing way. I loved this crazy code. I just love code. I think no matter how bad it is, it's one of those things that code is a medium to put

information down that is unlike writing. You can get a sense from this application, like from my blog post, what a medium to put information down that like is unlike writing. You know, you, you can get a sense from this application, like from my blog post, what writing code in this code base was like, but like you said, like go and do it for a month.

It's fun. It's interesting. And I think there's so much more code out there and so many different ways of writing code that we just haven't really unlocked yet that I, you know, I want to see us do more of. Yeah. Well, our friends over at Speakeasy have the complete platform for API developer experience.

They can generate SDKs, Terraform providers, API testing, docs, and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with Pydantic models for request and response objects and HTTPX client for async and synchronous method calls

and support for server sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at speakeasy.com slash Python. Again, speakeasy.com slash Python. And I'm also here with Todd Kaufman, CEO of TestDouble. TestDouble.com. You may know TestDouble from our good friend, Justin Searles. So Todd, on your homepage, I see an awesome quote from Eileen

Nucatel. She says, quote, hot take, just have TestDouble build all your stuff, end quote. We did not pay Eileen for that quote, to be clear, but we do very much appreciate her sharing it. Yeah, we had the great fortune to work with Eileen and Aaron Patterson on the upgrade of GitHub's Ruby Rails framework. And that's a relatively complex problem. It's a very large system. There's a lot of engineers actively working on it at the same time that we were performing that upgrade. So being able to collaborate with them, achieve the outcome of getting them upgraded to the latest and greatest Ruby on Rails that has all of the security patches and everything that you would expect of the more modern versions of the framework, while still not holding their business backileen and Aaron because we obviously learned a lot. We were able to collaborate effectively with them. But to hear that they were delighted by the outcome as well is very humbling for sure.

Take me one layer deeper on this engagement. How many folks did you apply to this engagement? What was the objective? What did you do, et cetera? Yeah, I think we had between two and four people at any phase of the engagement. So we tend to run with relatively small teams. We do believe smaller teams tend to be more efficient and more productive. So wherever possible, we try to get by with as few people as we can. With this project, we were working directly with members

from GitHub as well. So there were full-time staff on GitHub who were collaborating with us day in, day out on the project. This was a fairly clear set of expectations. We wanted to get to Rails, I believe 5.2 at the time and Ruby 2.5. Don't hold me to those numbers, but we had clear expectations at the outset. So from there, it was just a matter of figuring out the process that we were going to pursue to get these upgrades done without having a sizable impact on their team. A lot of the consultants on the project had some experience doing Rails upgrades, maybe not at that scale at that point, but it was really exciting because we were able to kind of develop a process that we think is very consistent in allowing Rails upgrades to be done without like providing a lot of risk to the client.

So there's not a fear that, hey, we've missed something or, you know, this thing's going to fall over under scale. We do it very incrementally so that the team can, like I said, keep working on feature delivery without being impacted, but also so that we are very certain that we've covered all the bases and really got the system to a state where it's functionally equivalent to the last version, just on a newer version of Rails and Ruby. Very cool, Todd. I love it. Find out more about Test Double's software investment problem solvers at test double.com. That's test double.com. T E S T D O U B L E.com. It's very easy, especially when you come into a code base to to you know guilfoyle the thing in the sense of how i was talking about guilfoyle earlier where it's like some other who was dumb or incompetent or malicious did this but as you actually start to like work with it and talk

to people and learn about it okay there are politics and there are power struggles and stuff. There are things that happen because we're humans. But a lot of it is like they made the best decision they had at the time with the information that they had. And I'm staring at it with completely different perspective years later. And like that stuff is fun to learn. And you actually realize like, yeah, that duct tape was really reasonable considering all the things they considered.

I just don't know those things. And so I really do appreciate code from that perspective, especially legacy systems that are still powering businesses and bringing value to people is that we want to be smarter than everybody else. But those people just had different contexts lots of times that we just don't have. And you start to learn those contexts and it gives you a new appreciation for the code that you're looking at and that's cool yeah uh peter nauer of uh backus nauer form um so bnf uh peter nauer actually has this great

paper called uh programming as theory building and one of the things in there that i think is really interesting is he talks about code bases dying. And by that, he doesn't mean that the code isn't running anymore or no one makes changes on it. He means that the people who knew the original context of the code base are no longer there. They're no longer working on a dead code base that's no longer alive because they've completely lost the theory of the code base. Why is this here? Why put these lines in the way that they are? How do I make these kinds of changes? And he argues in there that you can't revitalize a dead code base. Once a code base dies, you can create a new theory about the code base,

but it's impossible to ever recover the original one. And I feel like I've, I've worked a lot on dead code bases where the people who, yeah, the people who knew that context once are long gone. Don't work at the company anymore. And in some ways, like this code base wasn't dead because of Munch.

Because Munch could always provide that little bit of context, breathe some life back into it. And had Munch not been there, the code base would have been completely dead. No one would have had any idea what was going on. Is Munch still there? I haven't checked. I don't know if he even has a LinkedIn or anything. Maybe he does, but probably hasn't updated it even if he did. So I'm not sure. I know the company now doesn't have that name anymore. I don't know if that, you know, I don't know if that code base is still running or not. You know, it, it might be, but it also might, because it was customer support and

sales that when, you know, you get a big merger like that, that's one of the things that often gets, like they just choose one, right? Credit card processing code, I'm sure, is still running. But customer support might not be. Code base might literally be dead now. Merchants might be gone. Merchants three, if they ever got to that point, right?

Might not have any more data in it. I don't know. Yeah. I do like the way that you compliment it though. You talk about the beautiful mess, that section there where you talk about, which is kind of what you're talking about here, where there were no overarching design systems to work in. There was a lot of freedom. There was no documented design system. You mentioned. There was a lot of freedom. There was no documented design system. You mentioned that there was any concerns of co-duplication were out the window.

You can sort of carve out your own little section because trying to fix the big mess was impossible. So you just gave up and just worked in your own little world of insanity, as you say here. I think that's kind of cool that somehow, someway in this mess you found beauty yeah to enjoy i i think it's something that we should do more of you know i i've worked in systems since that are a mess but everyone's always trying to fix it and like i

get that impulse because i hate the mess too. But sometimes it's just beyond fixing. This is why people give the advice of don't do the big rewrite. Because it's much bigger than you think it's going to be and it's probably going to fail. I think the same could be said of trying to make sense of a 10-year-old, hundreds of thousands of lines code base, there's just no way you're going to be able to hold it in your head. And when you try to come up with some overarching scheme of what the code base is really doing, and now I can come up with the perfect abstraction and

it will do all of this, I just think it's a losing prospect, right? It's just not going to work. And so, yeah, I think we should embrace more, even in good code bases. I think we often want uniformity. We want everyone to have the same coding style. We want everyone to follow the same coding rules. And I've found that often, to me, that ends up causing more problems in the long run. Because once you have to have everything consistent,

as soon as there's a big change that needs to be done, it actually becomes harder. Because you can't do the big change all at once, and if you require consistency, it's harder to do those small changes each time. Even the connection to the users is interesting where you put in the after section where you describe it as a

ragtag of juniors essentially all the serious senior folks have have gone away even like we mentioned this is a game it's kind of interesting to think about this like as a thought experiment of this ragtag of juniors just like figuring it out talking directly to the users talking directly to the support rep who's got the problem, how can I make your life better? To me, that's like, in the grand scheme of things, it seems kind of interesting. Because I said, hey, why'd you stay there? And you're like, well, because.

And I think I can kind of see that light now in the after section. Yeah, and the next job I took actually had the exact opposite thing where we were actually completely forbidden from ever talking to our users, not just as developers, as a company. I worked at a big, a company that was, it's a big private company. They buy a bunch of companies and we were building an application for a sister company in this big group. And they hired a contractor to be the go-between. And they were scared of us ever talking to end users because they were worried those end users would think

that their job was in jeopardy if we were rewriting this application. So we were never allowed to talk to them. And I worked at that company for a little while, and then I left and then went to a startup that didn't do very well and i came back as a contractor at the original company and at that point they had changed this rule where they now were allowed to talk to their users indirectly not directly how so well okay so there were actually now users who were in the exact same building two doors down from where the developers were but they couldn't go talk to them.

What they could do was those people would write three by five cards of feedback and they would post them on the wall. And I came back as a contractor and I was so interested to see what is the feedback on the system that I knew was not a great system. It was a completely greenfield, brand new system. And because of organizational issues, as you can imagine, it did not meet user needs. And I looked at a few of them. They were all like, please fix this UI element. Please do these things. But one of them will always stand out to me.

And it was very simply put, it was, remember, you're supposed to make our lives better, not worse. Wow. That one hits hard, doesn't it? Yeah, yeah. And that's what this whole, that's what I had done. Like, the whole application I had been building with this team of 30 people

was making these users' lives worse. That sucks. with this team of 30 people was making these users lives worse that sucks i mean that's that's not how it's supposed to work yeah yeah and so like that's what i you know looked at like the contrast between this first job where yeah things were an absolute mess whereas there i mean there was no tests, right? Like they just literally didn't exist of any sort, not unit tests, not integration tests, not anything. We had manual QA that sometimes did some things. I had one really good QA for a while. She was fantastic. But like, you know, by all standards, it was wrong. Whereas the next job, by all standards, it was supposed to be good. It was right. You know, it was, you know, yes, everyone has different opinions, but it was spring and it was

angular and it, we had a hundred percent test coverage. That was the rule. Every line, every, if everything had perfect test coverage, we had a end-to-end test coverage suite, we had dedicated technical QA that did all of this, and yet it was a way worse code base. It was plagued with constant problems. It didn't meet the company needs. It didn't meet the end-user needs. It couldn't scale. Everything about it was wrong,

despite on paper, we followed all the best practices. Obviously, 100% test coverage is not actually a good metric i i know that well i did get that one changed to 70 at some point but i guess what's your broad takeaway from that circumstance i i think that one of the things that i've like come back to over and over again in my career looking And I think this first job really did teach me this. As programmers, it's very easy to give up on our responsibility and say, I'm just doing what the business wants me to do, and that's what my job is.

My job is to do what I'm told. It's to complete this story. Yes, I can maybe sometimes give input, but ultimately, I make it happen, but I don't decide what we do. I'm the how, not the what. And I think that that's always been the problem I've seen at these companies when things went poorly, was when developers just kind of gave up on doing what was good and what was right for the system and for their end users and for the code in order to just do what they're told. And I think that as programmers, we have to

accept the fact that we're not hired to do as we're told. Otherwise, we just wouldn't have the salaries we do. They wouldn't pay us this much just to not want our opinion. And even if they say they don't really want our opinion, we got to do what's right. If you're a good friend with somebody, you don't always do what they ask you to do, but you always do what's right for them. And that's how I think that we have to approach these things. And every company I've been at where the software made people's lives worse and not better, programmers have kind of given up on doing what was right and just did what they were told.

Well said, Jimmy. Well said. Anything else? Any stone that we've left unturned? I'm sure there'd be dragons elsewhere, but anything you'd like to highlight before we call it a show? No, I mean, yeah, there's tons of other stories. There's tons of other craziness, like the time I had to split the code base in half and all sorts of weird things like that.

But they're a little long. Yeah, fair. Fair, well, great, best, worst blog post. Love that you wrote that up. I think there's a reason it was popular. I think because it's a shared experience, well-documented, and it's fun to laugh at these things. We laugh so we won't cry.

Or maybe we laugh at you and not with you, because I didn't have that particular problem. But I appreciate you coming on the show and talking to us about it and for giving us some big-picture ideas and hope for the future of coding and also for the current state of coding and code itself. And a love for it that I share, at at least even when i despise it sometimes i still love it and i think that that's just the way it is sometimes adam i agree i agree all right you never know when you're the best of times

that's right right this seemed like from the outset like maybe not the best of times. That's right. Right? This seemed like from the outset, like maybe not the best of times, but actually it kind of was. Absolutely. Are you talking about our podcast? Are you talking about his experience? His experience. I know. His experience. No, it was fun, Jimmy.

Thanks for sharing. Thanks for going, I think, deeper than maybe is necessary to share a story that may not even matter to anyone else besides you. And you just shared it with the world. And now we can reflect on some of those key attributes that really reflect back on a good time. Really. It's cool. Thanks. I enjoyed it.

Yeah. All right. Bye, y'all. So Jimmy looks back on this time, beautiful mess as he said the best worst code base as one of the greatest times in his life as a programmer it's kind of funny how you can look back on times when you're in the moment and things kind of suck or they're not seemingly so awesome and it's actually one of the best times of your life. It's kind of funny, right?

As one wise person has told me before, these are the days. Okay, so that was a fun show with Jimmy. Very fun adventure. Very deep context. We got to dive into. I hope you enjoyed the show today. Make sure you check out Jimmy's podcast, Future of Coding. You can find that at futureofcoding.org.

So I want to mention this here in the post show because we're still not quite there yet, but we have definitely dove deep into the Zulip rabbit hole, if that's even a thing. I don't know. Maybe there's an analogy there I could have came up with, but I didn't. But we love Slack for many years. And then recently we tried Zulip instead. And so if you're already in Slack, there's an invite there in the main channel you can take advantage of. To come over to Zulip and to test the waters, many of the folks who have done so already have said, never go back to Slack, that Zulip is the best.

And I think that's how we feel we've talked about it on a recent episode of changelog and friends jared and i but i want to invite you go to the show notes there's a link in the show notes for you to join our zulip everyone is welcome there are no imposters hope to you there. We had some really awesome sponsors for today's episode. Assembly AI, the leader in speech AI. Check them out. Assembly AI dot com. Our friends over at Superbase. Check them out at Superbase dot com. Our friends over at Speakeasy. Check them out at Speakeasy.com. And of course,

our friends over at TestDouble. TestDouble.com. Just have TestDouble build all your software and you'll be good. There you go. Okay. Last but not least, our friends, our partners over at Fly. Yes, that's the home of ChangeLog.com. And that is the public cloud built for developers who ship. Deploy your app in five minutes at fly.io. Okay, those beats by Brake Massive Cylinder were banging. Gotta love BMC. That's it for today's show. We'll see you on Friday. Game on.