Python Bytes - #383 Why aren’t devs shipping faster?

Episode Date: May 14, 2024

Topics covered in this episode: I asked 100 devs why they aren’t shipping faster. Here’s what I learned Python 3.13.0 beta 1 released A theme editor for JupyterLab rich-argparse Extras Joke ...See the full show notes for this episode on the website at pythonbytes.fm/383

Transcript
Discussion (0)
Starting point is 00:00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 383, recorded May 14th, 2024. And I am Brian Ocken. And I am Michael Kennedy. And this, today's episode is sponsored by a new sponsor, MailTrap. Thank you, MailTrap. Listen to their spot later in the show. And if you want to connect to us, we're all on Fostodon and on Mastodon. And those links are all in the show notes, of course.
Starting point is 00:00:30 And also in the show notes or just at pythonbytes.fm, you can find out information to join us live on YouTube. It's always fun. And if not, that's okay. But you can catch us later. If you sign up to be a friend of the show, you can get the email. And the friend of the show emails are going well.
Starting point is 00:00:47 It's growing well. I'm excited about that. And also the open rate is staying very high. So that's pretty exciting. So thank you. People must care. I think it's cool. I enjoy getting in myself, actually.
Starting point is 00:00:59 Yeah, I do too. So also I enjoy hearing from Michael about the exciting news in Python. So Michael, what do you got for us first? Brian, why aren't you shipping code faster? Come on. What's taking so long? You know that feeling when you start a new project and everything's going quick,
Starting point is 00:01:17 and then after a couple years, you're like, why do we have so many meetings to add a button? You know what I mean? Or something like that. Yeah, it's it's tricky and so we're not the only ones to wonder this dax gupta said i was also wondering this but he took more action than most people i imagine would and said i'm going to ask a hundred developers why they aren't shipping code faster and then write it up. So I read it up and let's share it. There's some interesting things in here, basically.
Starting point is 00:01:49 Like, look, I talked to people from a lot of the tech companies you would know, Meta, Pinterest, Heroku, Roblox, et cetera, et cetera. Some of that I could probably make a joke out of, but I'm going to try not to. I'm going to try not to. But I thought it might be fun to summarize this for you. All right. So they broke it down into multiple big categories like meetings or human interaction type stuff. And the first big category is code bases. So code. And right at the top is tendency bugs. So you kind of got to look at these a little bit of what are your current today blockers? And maybe the person who said I'm blocked by this is a 27-year-old who never worked
Starting point is 00:02:36 when your only resource was a book and there were no packages, no dependencies. You want a dependency? You crack it open and you get the algorithm you start writing you know what i mean like that so so the problem is they say look if i hit what they've got a bunch of quotes that are actually nice in here like so jack an ex microsoft person says the number one barrier to shipping faster is dependency bugs so hitting a random mysterious bug with libraries that require require reading tons of old code or stack overflow links and GitHub issues. They're just trying to figure out, like, why does this thing, why did it stop working?
Starting point is 00:03:12 You know what I mean? And I would change that a little bit. Or maybe Michael's version is when something changes and it becomes incompatible. And the worst case scenario is, especially with the modern resolver in pipe, in pip, basically, is if we can't find a matching fit for you, you don't get it. Rather than, we're going to give you a warning that it's not a great idea, but we're going to let you run it, right? So if there's something that says, I have to have this version of a library less than a version, and something else says, I have to have this version of a library less than a version, and something else says,
Starting point is 00:03:46 I have to have this library greater than a version, and there's not an intersection there, you're now hunting around. And you're like, well, I guess we're done. And you can roll back, right? You could say, okay, well, what of this weird combination of things that it's coming up that I got to deal with, can I roll back so that there is an intersection?
Starting point is 00:04:04 But if you need a new feature, all right, then it's hard. Or even if it's not a new feature you need, it's just like, well, here's an hour and a half I spent juggling the intersection of these dependencies such that I didn't care about their interaction, really. I just wanted them both to be here. But there's this bizarre interaction I got to deal with. And my complaint about the pip being 100% like, no, we can't find it so you don't get it, is that incompatibility, the one thing that said less than, the other thing said greater than,
Starting point is 00:04:34 maybe that has to do with multi-processing. If you're not using multi-processing, it would probably be fine. Even though there's like a known intersection bug, if you're not using that part of the code path, it's not your problem. And you can say, like, I tested it. I don't care if it says it's the wrong version. I want to just keep working, right?
Starting point is 00:04:52 And so that's a lot of timing spent on that, right? Okay. There also is bugs in your dependencies. And I don't know if that was covered by this or not. Yes, that's true. That's true. But that's one of those, I think there's expectations that don't match really what reality should be. So like you said, we used to write all of our code. And even if you got it out of a book, you wouldn't blame the book author if there was a bug in it.
Starting point is 00:05:20 You would fix it yourself. But now we've got tons of dependencies and there's a lot of projects that's that are really 10 to 20 new code and the rest of it we're pulling in from everybody else um now if those other things have a bug do we think that it's we we just save time to not have to write all that do we go out and fix those in the dependent projects or do we just complain about it or yeah exactly exactly and it's usually open source usually prs are welcome right you don't want to do it but that's why i brought up like the the younger developer who didn't have to live through the time where there were no package or no package management things because there were no packages right you just maybe if you're lucky
Starting point is 00:06:02 you download some random code off the internet and you like work it into your app. Right? And so the velocity of writing software is so much faster than it used to be. And when it takes a step back by 20% because that thing has drawbacks, it's still vastly better than it used to be. So that was all I was saying there. Yeah. Yeah.
Starting point is 00:06:21 And I know you live through the same thing, right? Yeah. Yeah. Okay. So that's not the only one. No, no, there's plenty more complicated code bases. It's, it's tricky. I would have put this at number one if it was me. Yeah. So it says here, look, the law, the logic is sound. You've got a growing startup. You got to ship faster. You're going to die. No time to write docs. Early engineers churn out as the company grows. New engineers come into a big mess and they've got to deal with it, but they still got to grow fast, ship fast, et cetera, right? It says Maria from Amazon says, there's so much
Starting point is 00:06:56 undocumented in our service, including poor records of new features, non-existent or outdated info in our dependencies, or even essential things like best practices for testing, a lot of time is wasted in things trying to find what this is all about, right? And nobody has time to write documentation, which creates a vicious cycle. This is your number one? The complicated code base would definitely be number one. But the documentation and comments part, I used to have a tool handy that would strip out, do a fork of a repo and strip out all the comments.
Starting point is 00:07:31 I love it. Because it was so much easier to navigate some code bases without the comments. Yeah. Comments are great, but if it's, and I don't really know what else you do for this, but if the function is, let's say, 10 lines, but the documentation is 20 or 30,
Starting point is 00:07:48 all of a sudden you're like, okay, I'm trying to see. You know what I mean? It's like a little tricky. And you can do stuff, for example, in editors, you can code fold the comments and things like that, right? That could help. Yeah, but also, like, when you fix a bug, do you go back and fix the comment?
Starting point is 00:08:04 Sometimes not. Sometimes not. Okay, let's move on. help yeah but also like when you fix a bug do you go back and fix the comment um sometimes not sometimes not okay let's move on all right next major section of process and uh qa loops i can see what you you think about this but it's not so much uh test or no test but when you have a say a qa department and so on um it says taylor who's worked for a series of high growth startups, says, me, creating a test spec for QA, QA finding problems, getting a list of problems
Starting point is 00:08:30 two days later, fixing the merge conflicts because the code has changed since then, switching back, back to QA, keep looping, that kind of thing, right?
Starting point is 00:08:40 Maybe the challenge here is depending on other people for your code quality? Yeah, I think that's always in large projects always going to be a thing but having um one of the things that helps with that i know we're not trying to fix this here but um having embedded qa developers in the development teams but yeah that's an interesting suggestion for sure uh waiting for specs on larger companies you know just waiting for what are we going to build well i
Starting point is 00:09:05 gotta wait for people to prove you know the schemas or the api employees or whatever these are people wishing we had waterfall again i don't know what's going on yeah exactly just design it all up front and give it to me i think a lot of the next one also is awaiting stakeholder approval i think there's just a lot of that kind of stuff. We're a pure vegetarian team, so we have no stakeholders. Put the steak down. Okay. Raj from AWS said, Amazon meetings, approval,
Starting point is 00:09:39 taking 10 different stakeholders, talking to 10 different stakeholders because changing the color of a button affects 15 microservices. Oh, microservices. Could that just be its own thing like microservices? Okay, let's go. Writing tests.
Starting point is 00:09:55 Devs complain around tests could be basically divided into one not enough test or bad tests. So Grant at FinTech Unicorn. The biggest thing was we didn't have good tests or bad tests so grant at a fintech unicorn the biggest thing was we didn't have good tests or good types so i had to do a whole lot of work to do and testing of stuff whenever i wanted to ship things yeah yeah you could write them he could wrote them yeah yeah another one has to do with it doesn't at first sound like bad tests, but CICD taking so long. So running pipelines for shipping takes a lot of time.
Starting point is 00:10:29 And it ensures proper code coverage. And tests sometimes require these pipelines to take their time, which slows us down. Yeah, there's more. I know there's more here, but one of the metrics that I love is throughput. So is it throughput or cycle time? I don't know. I think it's cycle time. Basically, how long does it take you
Starting point is 00:10:50 to get from a single line code change that's to shipping? And how all that process through the CI pipeline, the testing, approvals, whatever you have to do to measure that and try to get that as low as possible and get the junk out of it? Because that is a problem yeah absolutely all right i can also blend the last one together so people are waiting for prs to be approved scope creep josh says the human tendency to stuff last minute items into the crevices of their luggage minutes before leaving for the airport manifests itself at software companies as scope creep slowly and surely it will push back your release date with every incremental addition feeling like an insignificant task but in aggregate adding
Starting point is 00:11:37 significant molasses to the team's velocity yes yes just ship it and then add those features later. Excessive meetings, I agree. I'm going to wrap it up with this one, Brian, because it's been kind of long, but this is interesting. Motivation. Diane, a former engineer at Meta, said the number one reason that slowed her down was motivation. So honest answer is I was working on ads,
Starting point is 00:12:04 and that's a very old, complicated, and large stack. I didn't understand it. My friends on younger teams seemed happier. I was miserable. Yeah, like, so basically, shouldn't be surprising. People want to work on inspiring projects. And when you're inspired, think more about it. You put more time into it. Well, could we squeeze a percent of a percent out of this retargeting thing to squeeze a
Starting point is 00:12:24 bit more money for the company like not the same as sort of inspiration is it all right well people can check this list out and hopefully you know maybe bring some of these ideas together to push back some of these problems at your company yeah yeah all right back to you brian well i want to talk about Python 3.13. Yay! Speaking of shipping things. Yeah, shipping things. 3.13, 3.13.0, beta 1 is released as of May 8th. I guess this is like a week old. Anyway, beta 1 released, and there's some fun stuff in here.
Starting point is 00:13:01 So betas are available to play with. And reminder, the beta does not mean that it's completely done. It is half baked. Um, so, uh, the, uh,
Starting point is 00:13:11 feature freeze for Python, is it an feature freeze? I guess. Is it? I believe so. Oh yeah. I believe when they hit beta, they say what's here is here.
Starting point is 00:13:20 Yeah. Ideally, unless maybe they have to go to take something out because they can't get it to work in the end all right so disclaimer it is still under development um but it is uh there are four beta releases um uh previews planned for 313 and this so this is a good time to this as we bold strongly encourage uh maintainers of third-party projects to test with13. So time to turn on your or at least check to see if you can turn on your 313 pipelines and CI. So a whole bunch of fun new things in
Starting point is 00:13:54 313. A lot of these are sort of internal stuff that are that I'm excited about. But you know, end users maybe might not care about there's the experimental fred free threaded build mode uh with um to disable the gill that's kind of fun a preliminary experimental just in time compiler um that's kind of neat uh this is cool a cyclic garbage collect a garbage collector that is now incremental which should mean shorter pauses for collection and programs with a lot of objects and uh different version of my malik mimalik i don't know how to pronounce that um that uh yeah some some possible memory improvements which is kind of neat the thing that i'm really okay uh a couple other notes i'll get to the exciting thing that I'm excited about. There's a bunch of removals.
Starting point is 00:14:45 So I guess we're getting rid of CGI now, whatever. That's how you run the web. However well I run my website. And last week I did, I mentioned this last week, typing.typeis. Type is, I think I pronounced it type LS. It's IS. That makes more sense. I was looking at the wrong font, I think I pronounced it type LS. It's IS. That makes more sense. I was looking at the wrong font, I guess.
Starting point is 00:15:09 But some of the exciting bits are that we have a new improved REPL, or Interactive Interpreter. And this is so exciting that actually Trey Hunter wrote an article about it. So even like the URL for this is I have the URL hidden is like my favorite thing of Python 3.13. So what's cool about the REPL is there's a bunch of cool stuff. There's
Starting point is 00:15:37 it's colored prompt, which is fun. We like colors now. And then blue. There's you can exit now. you used to have to do it used to say like exit bracket bracket or exit what it'll exit yeah exit works now oh my gosh i was thinking of doing a pr to make this happen i'm just like why is this so clunky yay i mean i mean a pep just like we gotta like we gotta rally folks this exit this is worth 313 just to get exit to work um uh block level history really cool so um uh basically this is like some
Starting point is 00:16:14 of these niceties that you used to get from other sort of uh alternative ripples things like being able to edit the block and some little block history. It's really pretty cool to be able to go back and modify things. And pasting code works better. So if you copy some stuff out of the REPL and paste it in the REPL, it doesn't paste in the dot, dot, dots correctly.
Starting point is 00:16:37 You know, it does that right now. So neat. The downside, it's curses, so uh won't work correctly on doesn't currently work on windows so that's actually it's a pretty big deal but you know anyway some fun stuff why not rich why not rich rich and textual that would have been awesome if they'd done that right i mean i'm not joking like there's it works all cross-platform it's got some incredible ui stuff would have been cool uh yeah rich rich and textual don't all work everywhere either but you know but more places maybe maybe this is a to be done or maybe just you get the old terminal on windows but you know that's 50 of the user base right right? It's not like, well, there's one particular MicroPython
Starting point is 00:17:27 type of device that's like this odd $5 thing that goes into tractors that it won't work. It's like half of the user base. Well, Blaze recommends instead of exit we should just require people to do
Starting point is 00:17:44 colon Q. How do you exit the ruffle? You reboot your computer. Okay, next. All right. Well, should we talk about our lovely sponsor MailTrap? Yes, indeed. Although I would like to add two things real quick, and then I'll tell everyone about them.
Starting point is 00:18:01 One, this is very exciting. It's going to be interesting to see where the free-threaded Python stuff goes because that's another one of the big deals. But it's weird, right? You've got to have a separate build. It's not even a flag or something you can set, but it's going to be something else. But also the high-performance changes outside of this free-threaded thing
Starting point is 00:18:21 with a JIT. Excuse me? A JIT? Not just interpreted? This is great. So very interesting. And then a follow up, I apologize, I can't remember who said it in the comments,
Starting point is 00:18:32 but DORA, DevOps Research and Assessment Metrics, includes four key metrics divided into these areas, deployment frequency or lead time for changes to measure team velocity and change failure rate and time to restore service measure stability so that's pretty cool door metrics yeah all right how about how about we talk mail trap now okay this episode is sponsored by mail trap an email delivery platform that developers love an email sending solution with industry best analytics, SNTP and email API, as well as SDKs for major programming languages and 24 seven human support. Try for free at mailtrap.io.
Starting point is 00:19:14 Yeah, thank you Mailtrap for supporting the show. Awesome to have them on board. We'll be telling you about them on upcoming shows as well. All right. So if you do Jupiter notebooks, Brian, I mean, he's going to settle so he All right. So if you do Jupyter Notebooks, Brian, let me use a human saddle so we see some pictures. If you do Jupyter Notebooks or JupyterLab in particular, it's got the way it looks and so on. But what if you could have a nice theming system for it? So there's an extension I want to highlight,
Starting point is 00:19:41 JupyterLab Theme Editor, which is an extension to provide basically editing of colors, font families, font sizes of different elements in the user interface, right? Yeah. So pretty cool. You can just install it, and then you can open up this little sidebar,
Starting point is 00:19:56 and it lets you pick the layout color, the ax out color, the border color, the UI font size, the content font size. So one of the things that sometimes drives me crazy about UIs, you know, think about like Zoom. If you go to VS Code and you say Zoom, it zooms everything. Like what if I just want to zoom just the stuff I actually care about, like the text, and I don't want all the other crap to take up more space in my screen to make the text even harder to read. I want it to over there and just see. All right. So having different settings for like content size versus UI element size is really awesome. So there's a bunch of things like that. And yeah, you can basically pick all these different
Starting point is 00:20:35 colors that you like. You know, you want a style to be a light blue on dark blue or dark blue on light blue or purple or whatever you want. There's a bunch of color palettes you can pick from, which is pretty nice. And it's not a huge thing, but it's kind of interesting. So pip install Jupiter theme editor and off you go. Yeah. I want to,
Starting point is 00:20:54 I want to try to make it seem like, like a, like an early two thousands. Yeah. Where's like the blinking elements and the gradients. I want now. Yeah. Yeah.
Starting point is 00:21:07 So that's a quick one, but people can check it out if they use Jupiter lab. Nice. Yeah. All right. Well, I've got another quick one a little bit. I,
Starting point is 00:21:16 I, we were talking about using rich and terminal or rich and textual in the, it's for colors and stuff and love both of those projects. One of the things that and you can get colors and think nice, nice help output if you're using like typer click or some of the other extensions for different argument parsing. But a lot of people and I do too on simple projects, I just use art bars. works great for simple, simple things. It's nice to test however it's it's more black and white so today we've got rich argpars so rich argpars is is cool it's not so if if you're having
Starting point is 00:21:56 a command line uh program that you want to use colors of course you could use rich just use the colors but the help is auto generated well rich arg parse or it auto auto generates the format rich arg parse uh gets colors in those so um it's got um you know nice orange color orange and blue and white and green and some different great colors to just to to really high it's it's a lot easier to read the help text if it's in color so if you if you can add if you're okay with adding a dependency this is great and um it's also like really easy to add i was like is it possible this is so easy and so the how you add it there is a whole bunch of features so there's um there's a they're built
Starting point is 00:22:37 in formatters so rich arc bars defines different formatters to help with the help formatter or the raw description help formatter they're different types but the easiest thing to do to try it out is all you have to do is um uh import a rich help formatter and when you're uh instantiating your argument parser or you just pass in a rich help formatter as the formatter class. And that those two lines of code, that's all we did. So I, I wanted to try this out to see how, how fun colors were in your help. And I've got a little test project called test arg parse apps, basically a hello world app that uses arg parse and runs testing on it. And I'm like, how do I do that? And, and so the, the help for the, my little
Starting point is 00:23:22 program looks like this. It's just white, normal. And when I added just these two lines of code for rich arg parse, you just get a nice orange and blue. It's really much nicer. Yeah. Color in the terminal is, I feel it's kind of non-optional. I will say that it is one more dependency, though. And one of the reasons why I use arg parse a lot of times
Starting point is 00:23:43 is because I don't want to add any dependencies and I still want argument parsing so keep that in mind so anyway that's a totally good point but i can imagine plenty of projects that said we're not going to have dependencies let's just use arg parse and then eventually they start having some and once you're already there you might as well throw this out here make it look nicer without writing much different code right yeah um and i love that it I mean, there are more features you go through, go look at the documentation, but I love projects where you can make your program better by just adding a couple lines of code. Pretty great. Indeed. Jeff out there likes to bring a full circle. So wasn't dependency issues one of
Starting point is 00:24:21 the biggest causes for slowdown development? Yes. Well, it's also, it's just two lines of code to change back. If you've got a test break, you can take that out. That's assuming that the original developer that knows that is still there. They could have left. No, I'm just kidding. That's a fair point, honestly. Anyway, I like it.
Starting point is 00:24:40 Well, those are our items. Do you have any extras for us? Only one. Only one. It's very exciting, for us? Only one. Okay. Only one. But it's very exciting. But it's just one of them. It's good.
Starting point is 00:24:50 It's the only one. We have a brand new course over at TalkPython. And this is done by Vincent, who's been on the podcast before with us. So it's called Getting Started with NLP and Spacey. Oh, neat. Yeah. So basically, if you have a lot of text and you want to process it, pull out entities, understand things about it, right?
Starting point is 00:25:11 Like at a real deep level, you know, NLP. Spacey is one of the most important ways that you can, one of the most important libraries for doing that. And so here's an awesome course that talks about how to do that with spaCy, but also even goes into using a little bit some of the extensions and fun things like spaCy LLM to integrate, say, ChatGPT or other LLMs along with the spaCy processing to pull things out. Talks about training custom spaCy models. Vincent did a really cool job where basically all the examples of this course
Starting point is 00:25:47 are based on TalkPython to me podcast transcripts. So he wrote a single function that will give you every line ever spoken on TalkPython over the last eight years as a generator, which is insanely awesome. And completely not useful very interesting right very interesting use of generators but then you can just start feeding that kind of information and say well you know who has talked about fast api and what when you do talk about fast api what other
Starting point is 00:26:18 projects in the libraries were talked about and like you could do things to discover sort of the zeitgeist of a topic or a tool over time by seeing how much other guests on the show talk about it and stuff. It's a really cool, really cool course. And I think people should definitely check it out. You can train and annotate your own data, do all sorts of fun things. So the link is in the show notes. And if you're on the list, I will probably be sending you an email about it as well. But yeah, very excited to have this new course up for everybody.
Starting point is 00:26:46 It's going to be a lot of fun. Yeah. The one extra I've got is last week I announced that I was going to switch the PyTest course, the one at pythontest.com. Of course, there is also a PyTest course at TalkPythonTraining. Definitely still worth trying out.
Starting point is 00:27:03 But there's the complete PyTest course that covers the entire Python or the pytest book um uh that switched to podia and that switch is done to podia so um well almost done it's there everybody's been invited um except for one person so they didn't get their invite i'll get that out later today but the um uh if you're still using the teachable version it's still it'll still be around until uh a little bit into june but please switch over soon um and and then the next step of this is uh podia comes with email services too so i'm switching from convertkit to podia for email and i'm gonna I'm starting that this week.
Starting point is 00:27:47 So doing that, that's the news on my end. And a lot of this has to do with just, since I'm not as a turbo of a course creator as Michael is, it's to keep my overhead down and so that I can do smaller courses cheaper and get more of them out to you. There's still good quality stuff it's just awesome yeah you keep the optionality right keep the optionality open so yeah all right yeah congratulations i know that's a big deal to move all that stuff around yeah it was a little
Starting point is 00:28:16 stressful i was worried that tons of people are going to be mad but um i haven't had any any complaints so far and i'm not inviting other people to complain, but you want to use a complaint now? Just kidding. Anyway. Oh, one of the cool things that it has that I didn't, I hadn't used before is there's comments within the course. So people can ask questions like right there. I know it's a complicated topic, so it's kind of cool to have comments. Yeah, that is cool. Anyway, do you have anything funny for us, Michael?
Starting point is 00:28:44 I do. I Michael? I do. I do. I do. Let's see. Let me put it up here. So another testing one to follow up with your extra, of course. Comes to us from the venerable XKCD. This is 2928.
Starting point is 00:28:58 Okay. So it's called Software Testing Day. Did you know there's a Software Testing Day, Brian? No. Yeah. It's right behind Valentine's Day, apparently. So do you and the other QA engineers have any fun plans for the holiday? Yeah, assuming the schedule system doesn't crash.
Starting point is 00:29:15 Software Testing Day is a holiday celebrated every negative one years on January 0th at 2571 p.m. That's why I don't celebrate then if you look at the hover as you do on xkcd it says the company tried to document how often employees were celebrating software testing day but their record keeping system kept mysteriously crashing this is good it's a subtle one right yeah celebrate every negative one years. I actually had the question today from somebody as to why we can't structure our tests so that we can completely test our complicated software so that we never see anything after we write the code. Yeah. What a messy place. I wanted to ask him how much time he had because proving software is uh correct is a something we tried to do a long time ago and it's not worth going down that road yeah
Starting point is 00:30:13 the only few places it might make sense is like the voyager probe or something like that or you know we're sending it out and we want it to run 50 years from now so no thoroughly testing is good but you still can't prove that you've caught all the bugs you just can't so yeah absolutely all right well yeah here's here's a nice little test case for them to use yeah every negative one years indeed nice all right well um oh fun cool blaze just uh mentioned speaking of testing uh blaze tube says fun fact right after this session they're meeting to go through chapter six of the podcast book yay anyway awesome cool well thanks a lot for this wonderful episode michael yep well as always bye brian bye everyone

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.