Programming Throwdown - 123 - Project Planning

Episode Date: December 15, 2021

How do you stay focused when working on large projects that span many months?  In this duo episode, we talk about Project Planning techniques and trends!  We also cover solving personal data storage problems and building CNC machines & printers. 00:00:15 Introduction00:01:33 UML00:05:22 Home NAS and other personal storage solutions00:18:09 Homebrew CNC machine00:29:37 Raft (Consensus Algorithm)00:36:54 The Mathematics of 204800:45:44 Book of the Show00:45:57 Manager Tools 00:49:10 Make Magazine00:57:50 Tool of the Show00:57:51 Workflowy00:59:10 GitHub Desktop01:01:00 Project Planning01:22:11 FarewellsResources mentioned in this episode:Tools:Workflowy: https://workflowy.com/b/Github Desktop: https://desktop.github.com/Companies:Manager Tools: https://www.manager-tools.com/Make Magazine: https://makezine.com/Other references: QT Designer: https://www.qt.io/Shapeoko: https://carbide3d.com/shapeoko/Curves and Surfaces by Bartosz Ciechanowski: https://ciechanow.ski/curves-and-surfaces/Inkscape: https://inkscape.org/Raft: https://raft.github.io/If you’ve enjoyed this episode, you can listen to more on Programming Throwdown’s website: https://www.programmingthrowdown.com/Reach out to us via email: [e-mail] can also follow Programming Throwdown on Facebook | Apple Podcasts | Spotify | Player.FM Join the discussion on our DiscordHelp support Programming Throwdown through our Patreon ★ Support this podcast on Patreon ★

Hey everybody, so we are here having another duo episode to talk about project planning, which is something that is super important. I think that there might be a dedicated class. I think at my college, there was a class called software engineering, but it was an elective and I never took it. But apparently, if you take, if you had taken that class, they actually cover project planning, but I had zero experience with project planning, you know, academic experience. And so, you know, it would have been awesome for

me to have this episode or something like it. When I started, I feel like, um, um, it's something that, that is super, super important to make sure that you kind of follow through and also like so that you're not just overwhelmed mentally with everything that you're able to sort of, you know, keep a lot of these notes. So you don't have to kind of have a brain dump, you know? Yeah, I think I also missed that class. Yeah, it's all right. We'll make up for it today. Actually, I remember the reason I didn't take it was because they spent a huge chunk of the semester talking about UML. And I looked up UML on the Internet and I was like, no, no, I don't think I don't think I want to take this class.

Oh, man. That rings right up there with i looked up the teacher ratings and the teacher is rated as hard so i uh no i'm just i didn't say i did that just i hear people say that yeah i think uml was and i don't even it's gonna be very lucid here but i think it was some kind of diagramming thing where i remember there's like a picture of a person and the person goes into a flow chart and then i guess a happier person comes out or something. I don't know that the person goes into the flowchart. But it was enough.

UML is interesting. We should talk about it at some point. I mean, I think it's like the idea that is it unified or universal modeling language? I mean, like the idea for describing architecture that way is useful. But I think it had a heyday probably, yeah, the mid 2000s the aughts is that what they were calling them the zero axes anyways they're on that time where they thought they were going to like auto generate code or like you would go into the box and like type your code in the box and then it would uh get hooked up yeah oh i remember those auto code actually for, for GUI stuff. Auto code is

still quite nice. Like I used, what did I use recently? Oh yeah. I used the QT, QT designer, which is, is, is a, for a project I was doing, I wanted a cross-platform UI and I mean, oh man, we should definitely do a show on cross-platform UI because I spent so much time researching this. And usually it's not like me. I mean, one of the things we tell people on the show is to pick something and just stick to it. But I really couldn't find anything really that I wanted that was cross-platform that worked well. And finally settled on Qt. Initially, I was a little hesitant to use Qt

because it seemed pretty heavyweight and it has sort of Qt as its own style and all of that. But from a programming standpoint, Qt was really nice. And they have this thing called Qt Designer where it, you know, it auto generates Python code and then you go and fill in like when you click the button, what should happen. So for that, it works well but i think that you know yeah back when uml was getting really popular i think people had this vision that everything would all programming would work that way where you'd have this visual designer and then you would write like little bits of code in the designer and that's never really taken off other than that one use case actually there was something uh patrick you might you might remember this there was something where

you could build like etl like data pipelines using this visual designer i think it's called a screwdriver or something but but you would like drag and drop blocks and then each block you would put in a bit of sql and then you would sort of like assemble these blocks of like sql and then each block you would put in a bit of SQL and then you would sort of like assemble these blocks of like SQL. And then one of the blocks could be, you know, right to this key value store or something like that. That actually was, I also thought that was pretty nice, but they ended up killing that project.

And I think what it was is the people who were much better at SQL and data engineering than I was hated it. So it was like, it was good for novices like me, but then somehow it was worse, a lot worse than just a text editor for the people who really know what they're doing. And maybe that's the general trend. Yeah, I don't know exactly what you're talking about. But it sounds vaguely familiar. I mean, I think there are places where specifically like the opposite is true, where describing the ETL pipeline that you describe in code with pictures as like an output documentation tool is still pretty widely used.

Yeah, yeah, that makes sense. Yeah. Cool. All right. So we're gonna dive into project planning. But first, let's talk a little bit about home NAS. So, you know, a lot of people have a lot of data at home now. I mean, I have, you know, a backup of all my photos. You know, I know Google Photos does, they change the resolution of the photo, right? So even though they'll store infinite, and actually, I think they're even, they're going to stop storing infinite. But anyways, so I just always keep a backup of all my photos and all sorts of other documents. I have a folder called Books. I have a folder called Research Papers.

It has like 90% content I've never read. It probably will never read, but it's there. We have a whole ton of information. We have Plex running, which Plex is a way to stream video within your house. And so all this data needs to be accessible and even ideally accessible outside your house with the right security, authentication, all of that. And what I used to do up until a week ago

was I used to just use the oldest desktop in the house. And so I had this desktop that I don't even know how old it is, maybe a decade old. And it has like a ton of hard drives in it. And it's barely able to run the latest Ubuntu, like it's just chugging along and trying to keep all of these hard drives up and running. And I had a drive failure. And so it really made me rethink, like, okay, what do I want to do here? What I ended up doing is similar. I ended up getting, there's this thing called a Yotta Master,

but it's basically a hardware RAID enclosure. So you plug USB into your computer and then the other end is this enclosure, which has, I think mine has five drives in it, and it does RAID 5 in hardware. So from your view, it's just like plugging in a USB drive, but it's actually this five drive monstrosity. And so you just get this like, I don't know, I think it like 12 terabyte usb drive is kind of how it looks to the computer and so i have that plugged into a raspberry pi so that's that's my new setup but uh but yeah what do you have patrick i have actually like an old proper was it synology nas that uh i got from someone um that didn't want anyone was like, oh, there's I've used that before.

And, you know, also, interestingly, and depending on your backup strategy, I mean, we have a similar thing, like backing up to the various cloud small parts of our data that's like super important, like, you know, pictures and documents and stuff. And then for larger stuff, what i've done between various computers is uh we've talked about on the show before i think it used to be bt sync but now it's resilient sync and so it it kind of mirrors over time between computers and even like some of the stuff that's like family important to like my brothers or my father or whatever we have one of those folders between computers so like some of the stuff that i put there and they put there as well and then it backs up multiple sites right

so like as long as if not both of our houses were taken out by some catastrophe in theory some of that data would be recoverable but also what i've come to realize is like a lot of stuff that i was backing up even if like hard drives are cheap it's just like it's not worth it it's kind of like storing stuff in your house like oh it's cheap to buy these clothes I'll just buy more and more clothes but then you need bigger closets and you run out of space and you start being compelled to buy a bigger house I think oh it is at Marie Kondo yeah that's right I don't know that story but yeah digital Marie Kondo I think the same thing is true like I don't need to back up my entire Steam library like whatever Steam has its own issue I don't wantie condo i think the same thing is true like i don't need to back up my entire steam

library like whatever steam has its own issue i don't want to even get into that but like if i have a steam game and the steam game is really big like part of having bought it on steam is the ability of like re-downloading it and you can get into your own like do you really own it or not but for me personally what i've just made that decision is like i don't need my all my games backed up because i can just redownload them if something were to happen and basically that can cut out a ton same thing like i don't need to back up core windows or even windows settings anymore like a lot of that stuff now the folders that have like documents in them yes but all the other stuff like settings like look worst case

i'll just build that back up myself and so what i do is i do make like for certain of those things like clones locally on that computer but for the most part like i just will reinstall and then bring back the data that matters and yeah that would kill a lot of time but it's the expectedness of it right like it would take a catastrophic hard drive failure which can happen and does happen but what is it going to happen during like my usage pattern it's like maybe once every x years five years four years and like reinstalling with to be honest is probably better than just like bringing forward all of that debris yeah yeah totally agree yeah that's a really good call out you know when i look at my nas i mean it has a lot of books i thought i would read and things like that but but i'm the same as you i don't put games on there i feel like anything i could read download

um you know i don't really have on there i mean i guess i could re-download the books and the research papers but it's more just like like uh i always feel like oh one of these days i'll i'll need to know like what are all the graph theory research papers i've ever looked up um so it's more of just like a personal archive yeah that's a great point though like i i feel similar and it's same thing with like my bookmarks like i constantly have not a good way to like bookmark things and like keep them organized it's just like right now i have like a billion tabs open and a tab is like a lightweight bookmark and there's like short-term long-term memory like substance short-term

occasion i want it to automatically go into long term and it's just because searching the internet for an obscure graph theory paper i don't read graph theory let's pretend i did like then you know searching is really quite difficult sometimes and this has happened to me like i don't know that it's important until kind of it's too late so having like a history of what you looked at or whatever and for like you books or whatever, what you're talking about is you want to save the curation of them, but like the data itself is not that interesting. But all of us fall into these traps

that like the tools just aren't there yet to model that. I mean, people will probably write us and tell us some good tips. Yeah, definitely. But I mean, I think like what you're saying is even just having a recording of the file names of your file folders, which is of course very lightweight, has a lot of the value of actually recording the data, but it's just easier to record the data. Yeah. So I think that the big reasons

for a NAS for me are pictures and videos. So, you know, we don't have, I don't think, well, actually I take it back. We definitely have some way to play DVDs in this house. I don't know what it is. I mean, it's probably in a box somewhere like an old PlayStation or something. Yeah. But, you know, we have all of the DVDs have been have been ripped, you know, even like the DVD of our wedding and everything. All of that is ripped and on the on the NAS and then and then all of our pictures are

on the NAS. If it wasn't for those two, then I would probably just run Resilio Sync on all the computers in the house. I mean, I have that as well. And so actually this is an interesting question is do you use Resilio Sync or Dropbox for personal things? I feel like I'm kind of using both. And sometimes I wonder like, yeah, I mean,

like is somebody gonna hack my dropbox and get forms with my address on them or something yeah i mean look people will get your social security number or your tax returns or whatever and commit very targeted fraud against you like i i mean it's the same thing like if this gets really depressing if someone decides to rob your house specifically they will watch it they will decide when you're not home. They will decide what's in it and what they're going to grab, right? And then to defend against that is very difficult. What you're looking for is defending against someone just like getting onto your computer

with some zero-day worm and encrypting your computer and then trying to blackmail you and a 1000 other people that like, you know, pay or we're going to keep it encrypted, right? Like, could they do that? Yeah. And then what you're talking about is unless they the NAS is good, because it's probably a completely different stack than your desktop. So the chance of it sharing a zero day, although recently, a lot of the NAS has had a really nasty vulnerability where they could get wiped. so i think like the synology ones or something so yeah so be careful i mean i think that yeah i think that dropbox like i totally agree with you that if someone's targeting you personally and for whatever reason they like just really want to get you and they're willing to fish

you and do all these things and you're you're in spot. But for all of us who are just normal dudes, I think that it's really the concern is that Dropbox somehow gets hacked in a way where everyone's Dropbox. But then I guess, you know, I guess to follow that train of thought, if everyone's Dropbox was leaked, my Dropbox is probably not high priority. Like in other words, like like seeing seeing forms, like, like medical forms from like a hospital bill with my address or something is not what they really want to see. They probably want, you know, I don't know, tax returns or something, but yeah. So, so, you know, I guess I have a mixture of Resilio sync, which is syncing among all the computers, similar to

Patrick. I have it also syncing with my folks' computers. And then I have the NAS for the pictures. We've just terabytes of pictures and DVDs and all of that. And yeah, it seems to work pretty well. I debated getting the Synology, but I felt like I already have several Raspberry Pis that run 24-7. I kind of felt like I might as well,

you know, installing Plex on Ubuntu is pretty, or on, I guess, Debian for Raspberry Pi is pretty easy. So I went that route, but I'd be curious, folks at home, or actually, Patrick, I mean, like, how would you think a Synology compares to just like running an Ubuntu machine? I mean, better and worse. It's a more appliance approach. So it's not the, I'm uploading everything to Google and subject to the whims of their terms and conditions or whatever. Like you'd normally, it's sort of your box.

So there's some level you could pull the hard drive off and like mount it on your own computer, worst case or whatever. A Boon Juizag is nice, but then you gotta like, there's time spent like updating it, choosing what to install, getting it to an, there's stuff but then you gotta like there's time spent like updating it choosing what to install getting it to and there's stuff like you know talk about like unraid or free nests or is it open nests which are more tailored to you know having a samba share so if you want to like mount it from windows or whatever which is stuff that sort of synology would come out of

the box you can figure all that out absolutely in fact, probably build a much more capable machine for cheaper unless you factored in how much time you spend on it and put any value on that time. Yeah, that's a really, really good call out. Yeah, I think as someone who'd already done a lot of Samba and all these things and that was incremental.

But yeah, you're right. If I was just starting out and I had a Linux machine and I needed Windows to see the hard drives and all of that, I mean, that is like weeks to get that right and actually have it be really smooth and work when the machine starts and everything. But that's not why. I mean, most of this is a hobby and it has a vague, in theory, help to our careers longer term by staying up on tech and learning new things and being in that space. So yeah, that's a good call. So maybe, okay, here's our advice. So if we were to, if we were to sort of conclude with advice for people who want to set up a home NAS, maybe you have a

desktop. So, you know, in my case, you know, I lost a hard drive in this machine and also the processor fan went out. And so the machine won't boot because it actually, it's interesting, it'll boot, but then it won't restart because the processor gets so hot that the BIOS won't let you continue. So like I had to shut it off, wait for the processor to cool down and then turn it back on. So I lost the hard drive. The processor is dead on this thing. Oh yeah, but I had a RAID. So it was RAID 1 or whatever. So I didn't actually lose any data. But if you have everything running just on your desktop and you have all the things you've ever programmed in your whole life, some of that stuff's sentimental or you might have pictures or other things. Definitely think about getting some kind of NAS setup. It could be as simple as

just putting more hard drives into your computer. If you don't have a lot of time, I think Synology, I know several people who have Synology and like it, but I definitely think something like that would be really important. You should check into it. If you want to put in the time and effort, I think getting a Ubuntu or Debian or getting a Linux box connected to a bunch of hard drives is, for me at least, it's been the way to go. Agreed. All right. So jumping into news, my first news story is this pretty cool thing. It's a homebrew CNC machine. Patrick, what does CNC stand for? I don't actually know. Computer Numerical Control. And so what that means to me, and maybe i'm totally wrong here is it's basically like a drill like a drill that you could control

with a computer and you could you can like carve things out and did i get that right how would you explain this no a mill it's normally a mill so it's normally like so the difference a drill is uh braced to only go like up and down, not side to side. So no side loading. So it has bearings meant to allow you to push into a material, but not drag through a material. Versus a router is a machine that is braced with bearings on the side. So if you drag it across the material and they're side loading,

that's like engineered for it. And so a mill or a CNC is, it used to be a mill was these things with like handles and screwdrives and like you would twist various knobs and move and shave stuff down and that's how most like heavy machinery is still built this cnc i assume you're talking about haven't looked at it yet is like a router that can go up and down but doesn't apply a lot of up and down force but then side to side is braced so that you can go through a material and like hog it out and it won't break the internals of either the thing that's moving the motor around or the motor itself. So you're saying if I, and this shows how little I understand

mechanical stuff. So you're saying if I took like a drill, like a power drill and I tried to go, well, I'd need a special bit bit let's say i had a bit that had blades on the side or something and i tried to like go sideways with the drill it would like break the drill because it's like not meant for not immediately okay but it would do damage yeah the basically like the bearings and the internal structure isn't meant for side loading got it so it's sturdy but if you did it, continue, it's more what happens is people try to do it on their drill press. So if you have a drill press and you're trying to move the material under the drill press constantly, and you give it too much, you're going to, it's not going to like catastrophically break. It's called a runout,

right? The amount of error there is in the concentricity of the spinning parts. So the spinning parts begin to wobble and the reason why is because you're putting pressure on a piece that isn't meant to take that load and so over time it deforms and then you get you know basically more error in the exact diameter of the thing that's spinning and when your computer controlling it there it's open loop so there's no closed loop which means it it tries to do a thing and it doesn't know if it accomplished the thing so that's open loop closed loop is you do a thing and then you take a measurement to see if you accomplish the thing you attempted to to change you attempted to

actuate so so this guy built his own cnc machine from scratch with like three parts and he got a uh there is like like inside of it all is this thing that you could buy i've actually seen it at home depot i think it's called a router or something but yeah yeah so so inside of all of this is like this cheap 30 router thing but then he's 3d printed all of this stuff around it and i think some of it you have to buy like you know timing belts and stuff and he's and and so he's end up with his own cnc machine which looks like pretty big i mean i looked into this at one point getting a tiny cnc machine is like at least a few hundred dollars and so with this it's like looks relatively cheap and you end up with a big cnc machine what's your take on this patrick

does this look like a good idea or just like a fun thing to do, but not a good idea? I mean, I've had one before. The one I had was a branded one, which the brand has sort of changed over time, called a Shape Oko. Oh, interesting. So it's more like a kit, but I'm looking at this now. Yes, that Makita router in the middle is meant for a human to hold and push through wood to shave wood sideways. So it's using the right got it these are like industrial ideas that are being made for the hobbyist if you have a need for so i had a shape oko which is the same idea as this when i had it when i was using it the

issue is that like it kicks up a ton of sawdust. So if you don't normally do woodworking or used to 3D printing, they're relatively, like 3D printing is relatively clean. When you start to do like power tool woodworking, this bit is spinning super, super fast and it's throwing tiny bits of wood dust. This picture I'm looking on their website

is they're carving MDF, which is basically sawdust glued together. So when you cut through it, it's remaking sawdust and blowing it into the air. I heard that MDF is toxic. It has formaldehyde in it. Yeah, right. And so, yeah, be careful with it.

But basically it's making sawdust that's going everywhere, tiny bits, and it's just flowing into the air. And so... So, wait, what do you do about that? Vacuum it up at the end. Oh, okay. Got it. Wear a dust mask.

Okay. A lot of people try to put a cover around it that has a vacuum attached to it. But even then, the vacuum fills up pretty quickly. It's a sort of constant fight. I would say there's one, I think it's called the mostly printed CNC MP CNC. That one I have heard about a little bit more. There's a shape OCO which is like a consumer

grade. There's several other consumer grade CNCs as well. I mean, it's if you have a use for it, like cutting out thick pieces of wood or making a sculpture in the face or whatever is definitely a super interesting hobby to be in. So yeah, I mean, it's cool. Yeah, well, check it out. I think if you have the time, it's similar to building your own NAS. If you want to develop a skill and you have some time,

this is a really cool project. I maybe would start with building your own 3D printer. I've done that before. And as you can hear, I know very little about this stuff. And I was able to pull that off. So maybe start there. But if you've done that, if you've built your own 3D printer, this would be a cool next thing to do. And I think the video is really compelling.

It does exactly what you'd want it to do. Yeah. I mean, the small size, I'm looking at Shapeoko now. They don't have the small one I had gotten. So they start much larger. But the ones that you used to make were four feet by eight feet. So they were meant to cut up like an entire sheet of plywood that you would get from Home Depot or whatever. But four feet by eight feet, most people don't have that much space in their garage.

And that probably costs like thousands of dollars, right? Yeah, that's right. Yeah. How much would this cost? I haven't looked at the BOM, the bill of materials. I mean, the big expense is the stepper motors and the sort of like rails and the controller. But I mean, if I had to guess, probably like $500, $600 if you had to source it news. Yeah, that's what I was going to guess too.

Again, cool hobby. So related actually unintentionally, but in a bit vaguely, related to a CNC machine is when CNC machines make cuts, you need to make curves through the wood to make your beautiful shaped electric guitar that you're carving out, or even a surface. And if you were to look into modeling that surface, you might run across something called NURBS or Bezier curves. Oh, I've heard of that. Yeah. So there was recently an article by, I am not going to attempt to say the person's name, Bartosz Cichnowski. There's a link in the show notes. There's a link in the show notes. That's right. It's called Curves and Surfaces.

And so he walks through and this guy's done a number of articles check out all the articles on his site he always has like a bunch of really really good interactive visualizations and so he starts through sort of like first principles of linear interpolations between two points and then what if you did that at like had two line segments segments that were, you know, of course, drawn by a linear interpolation between them. But what if you had, you know, instead of going between the points, you draw a line that connects an interpolation between the points and then a line that connects that line. And that's how you end up defining Bezier curves. Wow, this is cool. It's very interactive. Yes. And then if you start defining curves, then you could define surfaces and patches and, you know, provide smoothing on the derivatives, the first derivative or second derivative and what the impacts of that would be. And what would it mean to have kinks or, you know, sharp surfaces? And how would you handle that? goes it's actually rather if you make it through all of this and sort of really internalize all of it you will have a very good understanding of how a lot of modeling is done how a lot of 3d graphics is done this is so freaking cool you guys have got to go on this website i'm on it on my phone

here and even on the phone actually the phone is probably even better because of the touch screen but it is so freaking satisfying to play with these curves. So this is how like these curves that he ends up talking about is used like how Pixar like models the characters. Like if you want to model someone's nose, right? A nose isn't made of a bunch of triangles. A nose is made out of, you know, a sort of elastic material skin draped over a substructure of muscles and cartilage and whatever. Anyway, so you have to have an organic shape. And so that organic shape, you need a good way of sort of defining it. Eventually, you might end up reducing it to triangles.

But anyway, so this is really interesting. This comes up depending on, obviously, like what part of computer science you're in. But if you do much with graphics or rendering or even fonts how fonts are rendered fonts are defined using these bezier curves so how you define the like curve on the bottom of the s shape right like you define that in these bezier curves so most all fonts are defined using these style of things so it's actually used pretty heavily even if you kind of don't know it and you might run across it there have been recently a bunch more like very easy to understand interpretations of this. Prior to that,

you would have mostly gone to a computer graphics book and tried to read, and it would have given you a lot of math and a lot of linear algebra and said, here you go, here's how you define them. And if that's not your background, you would have been like, I don't get any of this. So explaining it in sort of layman's terms really helps you at least when you go into Inkscape and you're dragging the handles on the curve defining, you understand like what it's doing or why it loops back on itself, like at a fundamental level, like what is the computer trying to do? So I definitely recommend people sort of go and check it out. If nothing else, I'm like the cool graphics and learning a little bit more about like,

I don't know, like maybe fundamental is overstating it, but like a very like core piece of a lot of modern rendering and graphics and design. Yeah, totally. I mean, this is really, really fun to play with and it'll give you a really good understanding of, you know, yeah, how 3D geometry really works and it'll give you a really good understanding of you know yeah how 3d geometry really works and it'll connect the thing i love about this too is is it actually there's parts of it where it shows you an equation and as you are dragging these points around there's updating the equations in real time and so you can kind of in your brain you can like create synesthesia

between like this mathematics and then what's actually happening in this, in this picture. Super cool, man. Okay. So mine is also kind of a visual presentation, which is cool. We have amazing flow. This is, this might be our fastest flowing, cleanest flowing episode. Mine's a visual explanation of raft consensus protocol. So you have this issue where, you know, we were talking about your NAS at home and having RAID storage and so storing things on multiple hard drives. And that's not hard to do because you have this one USB cable or you have this one backplane on your motherboard, right? So there are all these hard drives are ultimately being controlled by one computer. But if you go to like, let's, a database at a really popular website, like take Amazon or something,

they can't just have one computer controlling everyone's shopping cart. And so they need to have this massively distributed, fault-tolerant database. But then they also have to have like many different, what they call leaders, you know, places where you can write data. And then also have many followers, places where you could, servers where you could read data that you can't write to them. And so, you know, you have this huge array of leaders and followers and, because it's so large, let's say it's like 10,000 machines. And if a hard drive lasts three years, then you could do the math.

But there's hard drives failing all the time in these machines. And there's other sorts of failures happening. And so you could have situations where even leaders fail. And now new leaders have to be elected. So imagine you expect to have three leaders. One of the leaders has failed. And so you now need to make one of your followers promoted to be a leader. All the followers are running the exact same code. You know, how do you do that? And, you know, if you could, you could imagine like heuristics, like, okay, let's have

everyone report their IP address and pick the lowest one or something like that. But any type of heuristic like that will cause you to have sort of catastrophic failures. Like imagine a subnet goes down and now it's like really confusing for everybody. And so someone a long time ago came up with Paxos and it's a way for machines to negotiate with each other without a lot of a priori information. And so eventually, you know, there is something that breaks ties. It could be as simple as just like the fact that the processors are not exactly doing the same thing and they're not deterministic. Right. So there's a way to break ties. But more importantly, there's a way to achieve consensus. So like once you have some way that ties are broken, can you use this tie breaking to eventually pick one single leader when all the followers raise their hand at the same time and say, I could do it. Right. And so, you know, Paxos is very, very difficult to sort of explain intuitively. And then there was Zookeeper, which is an Apache project, which dealt with some of the,

like Paxos is an academic thing, but then in practice, just like anything in practice, there's real world challenges that you have to face. And so Zookeeper was a project that was built to do this leader election and all of that distributed consensus. But that also became extremely complicated. And as Yahoo and Google and these big companies were finding issues, they were patching them in. And then maybe they would fix a problem that meant these 17 other patches aren't needed anymore, but they wouldn't get cleaned out. And it just became very bloated. So someone wrote Raft. And I don't actually remember where Raft came from. But Raft is kind of a rewrite of distributed consensus. It's very clean, very elegant. And what I'm linking to here is a really nice explanation

of Raft, you know, how it works and the principles behind it. And it has these cool infographics where it shows how, you know, okay, this leader dies and it represents everything with these little dots. So this dot went away and now these other dots, you know, need to pick a leader and it kind of goes through how all of that works. I think if you, when you start to think like you were kind of saying, it's actually a little bit mind blowing. Like when you think about if I have 10,000 computers, like trying to roughly mirror the state of each other and like decide what, what information is correct. And you start dealing with even the speed of light delays from one location to another data center you know halfway across the world and of course like it it isn't strictly just speed of light relative to the speed of a hard drive or whatever or ram

and then start thinking like you were saying just like power outages or whatever it isn't neutrino strikes or whatever you always like getting a bit flipped and ran to like, when you increase your exposure like that, the problem is just so much more complicated. Yeah, it's totally mind blowing. And like, yeah, I mean, I mean, all these leader, all these followers, they're all, you know, the same, they're probably even running the same hardware, right? So they all just raise their hand at exactly the same time and say, Oh, we saw the leader died, you know, I'll be the next leader. And it's like, well, what do you do with that? It's a scale of 10,000,

right? I mean, it's easy, dude. I mean, everyone always says the thing you're saying, like, pick the lowest, pick the lowest one. Well, pick the first person who raised their hand. It's like, well, how do you even define first like yeah like who is going to be the referee there if like once as soon as you have a referee who can define first then you've already solved the problem but like yeah you have to probably did it so yeah it's like now you have two problems you just have one god machine that sits somewhere in a like you know special housing that just you know guarantees to be up but what happens on sundays then yeah okay i mean mainframes people joke but like that that's like

banks and mainframes and stuff is in part like because of trying to get around this where they call like big iron right it's just like one really big powerful machine that you just four-way redundancy on every part inside and it it sounds like in some ways silly because things like google facebook whatever went to commodity hardware roughly but like there are cases if you're like transactions per second are low enough that maybe it just makes sense to have like one or two really really like a leader follower, you know, config. And the one machine just has an uptime that is like so guaranteed. And you restart it, like you said, maybe on Sundays when the bank is closed, like every Sunday during this hour, you restart the machine and the follower does the failover test. And then it goes down, you know, the next hour or whatever. And you have

like a team of people dedicated to even those two machines alive. Yeah. Yeah. I mean, that's definitely a real thing. I think even today there's companies that run that way and there's, there's nothing wrong with that. I think given, as you said, if the QPS is low enough, it saves a lot of hassle, but yeah, the RAF thing is totally mind blowing. Definitely check it out. I'm, I've always been fascinated by it. I haven't had a reason to implement it myself, but, but it's always been a fascination. And it's very, it's also, as I said, with Patrick's article, it's like super fun to watch the animations. I do not have a good segue.

So we're just hard cutting to my article, which is the mathematics of 2048. So there's more to the article title, but i don't want to spoil it yet so 2048 was a game that enjoyed i don't want to call it brief because someone will say like oh no i still play it's the best game ever but you have a a grid of oh man i don't even uh five by five i think it was it's sort of like five by five and then there are tiles that have a number on them so you start with like maybe the number one and the number one somewhere else in this five by five grid. And then you slide a direction to the right, up, down, or left.

I mean, that's each of the four directions. And all of the tiles slide to the side of the grid that you swiped to. So like one tile is in the bottom left and one tile is in the upper right. And you swipe right, then that bottom left tile is going to slide to the right side, is in the bottom left and one tile is in the upper right and you swipe right then that bottom left tile is going to slide to the right side just like the other one and then if two tiles that are currently the same number crash into each other they merge and become double that number so if they have to be consecutive or how does that work no they have to be the same number so like if you have a one and a one and you crash them into each other

they'll become a two but like what if there's another number in between the one yeah so then if there's like a two and a one and then a two and you slide down like stacked vertically and you slide down you'll just have two one two at the bottom of the screen got it so if you could somehow get the one to slide away then you could crash the two and two to each other got it i i probably butchered that but but you get i get it i get it okay and so there's of course like a bunch of descriptions of how you would um like how new tiles spawn so sometimes when you swipe new tiles will spawn on the screen and they spawn with like a given distribution and so like it's not always going to be it's most likely to be

i don't remember like most likely to be a one a little less likely to be a two a little less likely to be a four and so numbers are going to spawn on and the idea is to keep collapsing down until you have a tile which is 2048 so 2048 is what 11 doublings from one and wait now jason's counting that sounds right so 256 is eight right then 512 is nine and 1024 is yeah you got it right man that was fast okay so 24 is 11 doublings right uh so you need to keep doubling and doubling but each doubling requires two inputs right so if you think about it if you only ever ones, you would need to eventually have like collapsed 2048 ones together. It's a lot, right? That's a lot of collapsing you would need to do.

But what ends up happening is like Jason was already asking about, if you have like a two and then one and a two stacked on top of each other, like you really want to collapse the twos, but maybe you don't because maybe actually you're going to collapse that one into a two and you want to collapse it into one of the other twos to make a four. So when and how you have to manage the board. So even if it's, you don't just always greedily take the highest number or the lowest number, like you need a bit of strategy because the more the board starts to fill up with this new tiles new tiles generating it becomes too crowded

and then eventually you lose right normally without getting to 2048 well i see so if the board gets full up and you can't collapse anything then you're done yes that's right got it then you lose and so of course not every game is is winnable i don't think or at least not with every decision and so uh you kind of don't know what's going to happen. So you play and you just kind of mindlessly swipe, right? So then the question is like, what's a good strategy? So, you know, when you're playing, so in a game like tic-tac-toe, right?

Like the user, the other person can put in one of the nine spaces. So you can kind of like, I actually saw a t-shirt with it on like you can have a pictorial diagram of like the optimal tic-tac-toe strategy in like i don't know a very few states that you know each you only have x or o it's your turn then their turn they can only go in so many places the branching factor is really low you can actually represent the whole tree in like a very small amount of things which is like if you start in the upper left basically and then no matter where they play you can like predeterminately know the next state but then what about if you listen to people talking about like alpha go where the number of branching factors are like high

number of states is gigantically big there's a whole you know another class of problems so for this 2048 article the reason why i have it here is not for building general game AIs, but it's talking about something which I was hearing a lot about as you start hearing about machine learning, artificial intelligence, depending on your feeling about that, but is hidden Markov models and Markov decision processes, which is, I'm going to butcher this. Jason's going to tell me I'm wrong. But my takeaway from reading this article, which the reason I bothered to read it was because I

keep hearing about these MDPs, MDPs, you know, trying to figure out what you're going to do in this modeling is that for this game, as I described, first of all, there's this random part, right? It isn't like you battling another person. The game is just randomly generating what happens next.

It's like Sudoku, right? Oh, no, but it's not because when you swipe things... I've never played this game, so I'm just trying to piece it together. When you swipe things, there's empty space there. What happens? Like more numbers go in that space? Yes. So the game will roll some dice and decide, you know, generate a random number and decide

where and what number to place there with some probability okay yeah so you control your action but you don't control the other person's action it's not like the game is trying to beat you right it's just randomly happening so you can't even apply really game theory and that like the opponent is like i can know what they're going to do because strategically they're going to do this so each state you have a choice you can swipe left right up or down and depending on what the game is going to do will determine if that was a good or bad thing but that doesn't mean you just pick randomly you can know that hey i swipe up the game state is this the distribution is that the probability that that's a good move is, you know, point two, best move. And this other

move is point four, best move. And the way you might try to do that is by basically marching the states forward, figuring out like if you're in a winning or losing state, and then propagating it, you know, back to your state. So roughly, I'm giving you a high level vision of without going in a full explanation of Markov decision processes, because I'm no expert. But sometimes you'll end up in a state where you'll lose. And there may be a choice where no matter what you do, there's some probability of losing. But maybe you pick the one that minimizes that probability, right? And maybe there's a really good state you want the computer to give you. And so you want to try to increase the likelihood that you'll get an improvement. So he starts with, or she, I don't know, the article writer starts with a two by two grid, and I think getting to 64.

And then running this mark, this Markov decision process as a way of showing how this would happen in a in a sort of much more bounded problem set where you can sort of reason about this, and then talk about how you would extrapolate it to the full game into playing to 2048, where you may not be able to hold all possible game states in your memory at one time because it becomes too high. And how would you apply heuristics? How would this Markov decision process work? And just a really good introduction. So if you, in your ramblings about the internet, have come across Markov decision processes, which is related in a way that I'm entitled to speak about

hidden Markov models themselves, then this may be an article that is pretty interesting to you. Cool. Yeah. I mean, you did a really good job. So just like people don't know, that was my whole PhD was on MDPs and POMDPs and everything. And yeah, the article is super, super interesting. Definitely check it out. If you've played 2048, it's really cool to see something that you've played, you know, connect this tech to something that you've played.

It really like makes it real. But yeah, I think you nailed it. I think it's, you know, you could say that like, maybe it's a partially observable MDP because you don't know what's coming next. But you can model it with an MDP. And as you said, it just means that sometimes you'll lose anyways just because you're unlucky

and that's okay. And you just kind of model that into the system. Okay. Well, I'm glad that you reaffirming that I got a takeaway actually makes me pretty happy because that means that maybe I'm advancing. You never know, right? When it's a topic you don't understand, you read a thing and you're like, oh, I kind of get it. We don't know if you're right or wrong.

Yeah, I know. I think you nailed it, man. Yeah, I've always had a big fascination with puzzles and games and the way that they're solved. And so this kind of stuff definitely scratches that itch. Definitely check out this article. It's really solid. I think it's time for our book of the show. Book of the show. My book of the show is a podcast. I'm totally cheating, but I've been binging on this podcast. It's called manager tools podcast. You know, originally I, I found it through a literal search. I actually, I literally wanted I literally wanted tools to make me a better manager.

But what I found instead was their interpretation of tools, they really mean techniques or skills or advice. And so while I didn't get any literal tools out of this podcast that I could use to make me more efficient, I'm learning a ton of information. And I would recommend this to anybody. I mean, not just managers, like anyone who is working in a job or plans to have any job at all should check out this podcast. You know, I have a lot of these situations or things that I've had in the past. And, you know, some of these are lessons I learned the hard way and they kind of explain it in so much, you know, better detail and candor. Some of these things are things that

I was doing that I think I'm going to change. Some of the things they talk about are reaffirming things that I felt like were good ideas. But in general, it talks all about, you know, kind of being a leader. And I think they do a lot of talk about project planning, which is our topic today. So it's also a good reference there. And I was listening to an episode today. And one thing that really, you know, was funny was they were talking about sort of how do you delegate so imagine you have a giant project that you're leading and you need help with that project right you could be a manager and you're given a project with large scope you could be a

an engineer you could be just a high school or a college student and you have some team project that you're working on where you're trying to get help with other people. And it talks about, you know, how do you sort of share that responsibility? One of the funny things is they said that one of the people that they talked to said, oh, I wish I was CEO because then I wouldn't have any direct responsibility. And I think the idea there was like, if you're the CEO, you just tell people what to do. And then when you tell them what to do, now it's their responsibility. And, you know, I mean, most people, I think, could see where that's going. But just to like make it explicit, that doesn't work that way. When you ask someone for help, what you're doing is you're just doubling the responsibility. Now you're responsible and

that person's responsible, but it's necessary because you can't do everything by yourself. But you're not actually reducing any responsibility by becoming a leader. It doesn't work that way. That was a really funny thing that came up in today's episode of their show, but, but definitely check it out. I've been having a blast listening to it. The people have good candor. They're ex military people.

And so they, they speak like ex military people. And so I kind of enjoy it. It's kind of gruff and like straight to the point and stuff, which is that, that is I think kind of good for easy for me to listen to, but give it a give it a listen i've been binging on it i've been really enjoying it my book of the show is also not a book but it is print so i guess it counts it's make magazine you know i i'm i'm pretty sure that i probably had this like as a reference or

something on here before but i mean people probably had this like as a reference or something on here before. But I mean, people probably at this point have heard or seen Make Magazine. Yeah, it used to be like at Barnes & Noble and stuff. It might still be, right? I think it still is. I mean, if you ever go to those stores. Yeah, if Barnes & Noble even exists. I mean, I don't know.

It goes through ebbs and flow of like, I guess, popularity. You know, there was like the really big boom of like cosplaying and hobby stuff like when the internet was so in sympathy and like instructables was really huge i remember yeah i remember that yeah i mean it probably still is around i mean i know it is around i think it got bought by a company anyways you know this thing has become went from like the first making it big to like actually even though it was a niche magazine it became too generic and more niche magazines that were you know like hobbyist electronics hobbyist cosplay making hobbyist cnc like all those things became their own topics with enough you know ability for people to look

into them that you know maybe it sort of robbed some of the like make magazines you know yeah kind of shine but i always thought that you know although there, kind of shine. But I always thought that, you know, although there's been very few projects I've ever done out of make magazine, always appreciate that, like, they go through document describe and have a way of kind of covering a wide variety of things from like crazy dangerous, like Tesla coil making to like, know hey here's a air pressure uh you know soda bottle rocket you can make in your backyard and you know i just i still subscribe to it i even keep the magazines because i feel like having it in hard copy it was just cool oh you have the physical magazines like recent ones oh yeah yeah yeah yeah oh i would show people but we're only

audio right behind me. I'm looking right now and I'm seeing thousands. I'm just kidding. I don't know that I've kept everyone, but I keep them always like, oh, there's something in here I want to make or reference or look at. This is good. I don't know. Maybe I'm an old person now, but it's nostalgic.

No, I think it's kind of like the NAS, right? It's satisfying to just have it. Yeah, exactly. And just physically thumbing through it when you're on an airplane or whatever, where it's satisfying to just have it yeah exactly and you know just physically thumbing through it you know when you're on an airplane or whatever where it's still like electronics is a bit of a chore although that's even becoming less and less i kind of used to look forward to going in an airplane and not being allowed to use electronics i catch up on reading but now like that's not even a thing well not going on airplanes but also airplanes that allow your

electronics and having like yeah that's true you know what. Well, not going on airplanes, but also airplanes that allow your electronics and having one. Yeah, that's true. You know, what I've done now is I've put my phone in. Even when the airplane has internet, I still put my phone in airplane mode, you know, without the Wi-Fi. And that way it's like I'm at least unplugged and I won't be able to use the internet. But also the thing I'll recommend is like, you know, of course, they have a blog, lots of articles. I appreciate what they do.

They used to have, which I think have also sort of tapered down or even, you know, gone away to some extent, especially now with COVID. But they used to have maker fairs where, you know, people would come and just celebrate me. That was always really cool. Have you ever gone to that?

Yeah, I have. Yeah, yeah. I went to the one in the Bay Area and it was probably three years ago. It was super fun. I always wanted to go back and then I wasn't able to go the Bay Area. And that was probably three years ago. It was super fun. I always wanted to go back. And then I wasn't able to go the next year. And I think it rained.

And then the next year was COVID or whatever. So yeah, I don't think that happened. And then I think they've gone through some sort of restructuring. I didn't follow it exactly. The other thing I'll say is, you know, if you have other people who are peripherally interested, it may be very hard to spam them a link or convince them to go to a website or watch a YouTube video, or if you have children, right, or whatever, like having the magazine around and someone picking it up and just looking at it. And

like, you know, the sort of lower commitment of that and then becoming intrigued. You know, my kids call it the robot magazine, like, Oh, look at the robot magazine. Yeah, look at it. And I think, you know, just having that stuff laying around where you can sort of bump across it as opposed to your, you know, normal kind of filter bubble for maybe other people, even if you yourself, you know, could find it and look for it. Uh, there's some value to that anyways, but I, I appreciate what, you know, make magazine sort of has done in the influence it's had. Very cool, man. I, I kind of on a related note, I actually learned to solder. So I've soldered. Yeah, I've soldered things before. But usually

soldering to me was like I would just turn the temperature of the soldering iron way too hot. This is the other thing. I bought the soldering iron from Ali Express. And it does it didn't tell me Celsius versus Fahrenheit and I just assumed Fahrenheit being an American it was actually Celsius I was like man it's like so freaking hot and so so that was a big problem but then also I would basically just end up with this huge glob of was it flux I guess no solder or solder okay i don't know but just just and i've never had to solder something on a pcb before so i've never had to be that precise anyways it was usually just repairing like you know like my son pulled a plug out of the wall too hard or something so

like you didn't have to but i know but just this huge glob of metal like and you just be this giant like uh state puff marshmallow metal man just hanging off the end of everything and that was my way of fixing things but i i bought a a tiny speaker which i'm using for an arduino prod for arduino halloween project which which we did and um as someone who doesn't know how to solder i was like well i'll get the one where i can screw you know yeah like it has like a screw when you screw it puts two plates together and squeezes the wire terminal, screw terminal. There you go. Um, but the screw, the, the wire, the speaker was smaller than I expected and the wire was smaller than I expected and it wouldn't stay in the screw terminal. And I flipped the PCB over and there was like

contact points there that you could solder to. And I was like, okay, I rolled up my sleeves. Like I'm going to learn how to solder for real, a real like a like a professional solder oh professional solder okay yeah well i'm gonna try i'm gonna i'm gonna at least get to the amateur level and so uh you know it just took me a lot of practice and um and yeah i got it to where like i had this really nice shape and it was like also holding the speaker wire uh you know pretty well so i mean i will warn you like i assume the speaker wire was probably stranded and not solid and soldering stranded wire is much harder than solid wire and soldering wire in general is harder than soldering like actual components like resistors diodes you know

through components so yeah you didn't take on the easiest thing for sure yeah i mean it was just thrust on me right but yeah i mean i also yeah i was worried that one of the strands would you know cross over and short circuit um but that hasn't happened yet so uh i was so satisfied i did all that work and then um i started the arduino up and uh it was uh probably even an instructables site but it was a project where it would play like the mac boot up sound like that you know and it worked and like it came out of the speaker and i was like oh it felt so satisfying it was unbelievably satisfying to do that and then i looked around at like all the

other really terrible soldering jobs and i'm like like, one day I'm going to, I'm going to fix these. Yeah. It does wait till you learn about doing surface mount soldering with like 0.1 millimeter pitch, like pieces of pins between things. And so those people get crazy. Oh man, I can't even imagine. But then of course they also use like ovens to basically melt the solder and the pieces together in one go oh okay yeah yeah one thing i realized is i'll need some kind of i think they call it helping hands but it's like these hands with a magnifying glass alligator clips on them yeah yeah right yeah

also the trick with the wire while we're at it and then we can move on is get smiths is they call it pre-tinting the wire so if you twist the strands together right like just twist them just like one wires strands like all together so and then take your soldering gun and just apply solder only to the wire and then cut it to the length you want and then what happens is you have basically already solder on the wire solder on the pad so you just have to touch them together and briefly touch the iron the soldering iron to both of them and then the solder will flow together and the solder more readily flows to itself than it flows to either of the other metals and so you basically reduce the like time you have to spend

uh sort of trying to get that to work oh interesting very cool all right well in the time we'll we'll go a little quicker here we've had so many cool things to talk about but we'll we'll want to give time to the to the to the task at hand so my tool this show is workflowy this is very simple it's a recursive google doc so you know you can write a google doc or you can write a workflow you write a document and at some point in a document if you're doing something that involves planning or something, at some point, you're going to have a bulleted list. Like it's inevitable, right?

It's like things to do or what have you. Anytime you create a bulleted list in Workflowy, you know, you can have a nested list and you can actually collapse the list. And then if you tap on one of the bullets, it actually opens that bulleted list as like its own sub document. And so that way you don't have this enormous, massive bulleted list.

It's like with all the levels, like instead you have, you know, maybe five bullets, they're all collapsed. And when you tap one, it takes you straight to that, you know, it unfolds that in a new document. So the high level thing here is Workflowy. It's on desktop. It's on mobile. So it's on all the important platforms.

And I found it really, really useful in keeping track of stuff. I think even for project planning, I found Workflowy to be super useful. So I've used it for many, many different things. I highly recommend it. Nice. Is it on your I highly recommend it. Nice. Is it on your NAS?

Tie it all together. Mine is GitHub Desktop. I don't always use GitHub or Git. Well, I should say I do use Git for our workflow at work, but I am not very good at it. I often find myself Googling, how do you do this thing again? Yep.

GitHub Desktop, which if you uh have uh specifically if you're using a github at all or even if you're just i think it works even if you're just doing it locally the nice thing is that it's a a gui for the get to workflows so it allows you to visually see some of the things especially for me helpful if you get into more complex circumstances or rebasing and yeah some things like not going quite right i don't normally need it if i'm just doing you know git commit dash a dash m check in you know i don't need you know github desktop to do that for me but visualizing if i have a whole if i'm doing a refactor or a whole bunch of files are open

or I'm wanting to see something, right? I think it actually has been really nice and I'm appreciative that they, you know, kind of took time to make that. So why is it GitHub? GitHub desktop is what it's called. But is it called GitHub because they made it or? Yes.

Oh, okay, got it. So it's for any Git, it's just made by github you know i'm gonna assume it is but i actually don't know but yeah i mean if a lot of people for hobby projects are using of course like open public github repositories and so uh you know i still think it has use even for there um but yeah, being able to visualize this, if you have a GitHub workflow, and I think it's super valuable for a lot of those more complex things that really help by being visual. Yeah, I think it has extra features

if you're using a GitHub Git repo, but it works for regular Git as well. I also learned about some really powerful Git tooling, and then it required me to first learn Emacs. And so I failed, and this is my alternative. Oh, man. Poor Emacs. I used to use Emacs every single day, all the time. And now it's like Visual Studio Code just really is just so nice.

It's very hard to match that to convenience and the integration with the OS is just a lot nicer, which is a shame, shame for Emacs. All right, well, I think it's time to plan our projects. That's right, project planning. So what are different types of tropes like there's a lot of these like waterfall and agile and yeah i when we were preparing for the show jason and we're talking about this which is a whole other subtopic first of all you know this is by no means expert

and we're actually going to talk about these briefly and then not talk about them again and talk about more practical uh thoughtful, common ways of planning projects. But there are rigorous approaches to this process. And in fact, if you are, which I suspect most people are, of a software engineer inclination or not a product or project manager who regularly interacts with this, you will probably be more at home with what we're going to talk about. If your profession is managing projects, you will probably find this appalling. So I apologize. But, you know, the traditional way people kind of build software is, I guess you would call it waterfall. You do step one, you get it kind of done, you get your thing, then that flows into step two,

then more work is done, right? You flow into step three. And if you found a problem or issue, you would go back to some previous step and sort of start over. And then once you got to the end of your waterfall, your product would be built, right? People realized that was pretty inflexible,

that requirements change, that there are issues. Then really popular was the agile method where the, as the name applies, the idea was to be more agile, to sort of get something up and working and then sort of find the next bit to be added, the next feature to be added and work in smaller units of time. So you could get feedback from your customer. And I think that had a lot of really positive impacts on the industry. I think there's a lot of cases where maybe some of the assumptions about how you could

or couldn't get feedback and how long stuff would take, like there were adaptations of the agile method and maybe some people took it to be more of a stricture than even the originators intended it to be. And so it got its own sort of like, you know, stigma. But what I would say is at least in the people I work with in the domains I work with, most people fall into some sort of hybrid approach, definitely not a very rigorous, you know, the agile TM, you know, methodology, with the full scrum, you know, all that stuff, but definitely not the, you know, original waterfall, you know, with its, you know, strict representation. And I think there are uses for both, and important features are both, but most of the stuff that I see happening

falls kind of like somewhere in the middle with some hybrid practical adaptation of learnings from both. But I also will say that in the industry and current part of the industry I work in, that project planning is needed because we always need organization. But the interaction with the customer isn't a contractual thing where you would definitely, you know, want to make sure that your organization was such that it can be communicated externally. Yeah, I totally agree. I mean, I think Patrick and I both are in a similar position where we work on longer term projects that don't have a lot of like tight SLAs where there's someone, you know, someone files an issue and you have 24 hours to respond to it type thing.

So it's going to have that bent to it. But I think that, yeah, I think Patrick's totally right. I mean, you always have a waterfall in a sense because you want to accomplish things that, like you want to scaffold, like you want to build something and then you want the next thing to be derivative. But then at the same time, you want to be able to respond to the environment, which changes massively. You know, someone might write a research paper that scoops your idea and now you have to pivot, or they write a paper that causes you to have to change the way that you want to go about doing something. And so you have to constantly adapt to that. And so you need some of those agile methods as well so that, you know, there's a lot of different concerns. I mean, one is, you know, you adapted so many times at the end, it's really hard to know what you really did. give people this security that they're marching towards a goal, even when things are really erratic. I mean, that's what Agile is really good at. So one thing, no matter whether you choose one

of these approaches or you do a hybrid or you go your own route, there's going to be several different factors of project planning that are always going to come up in every plan. The first thing is headcount. So how many people or how many portions of people's time is dedicated to this project? The second one is going to be scope, right? Now, typically for a project, well, second one is going to be scope and the third one is going to be time, right?

And there's this famous triangle. It's like headcount, scope, and time, and you can only control two of those. So if you want things done quicker, you can add more people, or if you want things done quicker, you can reduce the scope. But if you're not willing to do either of those things,

you can't get things done quicker. Like once you've solidified two of those three points in that triangle, then the third one is automatically decided for you. So I guess diving into all three of these. So your headcount is relatively straightforward. I think where headcount gets more complicated is its relationship to the other two. So being able to answer at the margin and say, okay, you know, if we hired another person,

then X would happen, right? Then we would have X more scope, or we would be done this many weeks quicker. And so that's what makes the headcount part kind of interesting. Similar to, you know, answering these counterfactuals, you're also prioritizing so you can answer counterfactuals the other direction. If I took a person off this project, what would happen? And so that's, I think, what makes the headcount part, not just an accounting thing. I think also the tricky part about headcount, which kind of everyone knows, but everyone always forgets, is like Jason is saying, if I have one person and I add another person, I can do two times scope or half the time. Oh no.

Like, no, no, no. That's definitely not how it works. Yeah. And it's not even an equation, right? It really depends on the work. And sometimes you don't know when you're getting started, like, can you easily segment this work? What is the communications barriers?

And I think there's just so many nuances to that, but it's something I see that everyone forgets constantly. Yep. And also, I think that, you know, the teams are usually heterogeneous, right? And so we had a situation where we ended up with way more research scientists than engineers on the team for a portion. And so there were things that couldn't get done that were high priority, even though we had the headcount, but we didn't have enough of the right type of headcount. So that becomes an issue as well. And then even there's some things that,

you have to look at the team, there's some things that people just don't want to do. And so that becomes part of it too, right? So I think all these things make headcount a little more complicated, but I would say it's the simplest of the three. And for that, I've generally just used an Excel spreadsheet. What about you, Patrick?

What tool do you use for headcount planning? Nothing so formal. I don't know that it's always been, I guess, not one monolithic project where we brought people on. It was always, we have a variety of projects. And the question, kind of like you're saying to management, was like, if I could get you another person,

I could keep this up or take on more work. If you took a person away, please don't. Then, you know, we would not be able to do these things. And so it's never required. There's always been, I guess, at least for us, like a pretty hard constraint on getting new people. So it was always make the most

with what we have. And then occasionally you were successful enough that people wanted more of something. And then basically you were offered like, would it be useful to have another person? Yep. And then the answer was already kind of a given. Yes. Yeah. Yeah. I totally agree. Yeah. So I think for headcount planning, you don't need anything too fancy. I think just a spreadsheet or a Google Doc is more than enough. So the second one is scope. Now for scope, this is where I think having, you know, some set of work items or things that you could be doing. And you want to track those things. So what I've used in the past has been some variation of Kanban boards. So Kanban is basically you have, and I'm probably going to butcher this too, but you have, you know, different categories.

You have one category of things that, you know, we want to work on or things that we're going to work on, but we haven't got to yet, but they're imminent. Things that you're working on, things that are done. And then you have this other category called like the icebox, where it's, these are things that we haven't, you know, committed to working on them yet, but there are ideas that, you know, we've, so usually things get put in the icebox, they're not created in the icebox, right? And so, you know, most of my scope planning has followed along those lines. And then with each of these items, you'd want to assign some type of priority. So, again, answering that question, if somebody leaves, you know, what's the item you would kick out, you know, definitely would be the constraints notwithstanding, it would be the

lowest priority item. And if people join, you know, what things from the icebox could we bring and start working on? Yeah, that's effectively how it goes. The issue is people are constantly adding new things. If you're planning six months in advance, you know, halfway through that plan, there's a hundred different things that have been added. So there's a scope creep that you have to deal with. I guess I'll try to tackle the third one. And then maybe since it was yours, you can follow up with whatever I missed. But when we talk about time, I think you were mentioning, right, like this is a triangle. So if you have headcount and scope, then like, sort of the question is, okay, well, fine. Time is constrained, but what is the time going to be? And I'll call it here that for me, at least I find it very difficult for people to

remember the distinction between the number of hours a person can work the project versus what calendar date the project will be delivered on. And so people will say, oh, you know, you ask an engineer, how long would it take you to do this thing? And I'll say, oh, it takes me six hours. He's like, okay, well, cool. I'll put it down for you to have tomorrow. And it's like, well, no, no, that's not how that works. I'm not going to work six hours until next week or until these five dependencies come in. And so saying time is correct, but yeah, a little bit ambiguous.

And then also, when we talk about you've had scope and then now you say time so you just try to define the feature well but there's always variability to how long it's going to take to implement that feature and so you know people do oh it's t-shirt sizing is it a small medium or large uh you know project and you can take those approaches and try to roll them up, give them points, say the points don't matter. But then someone has a rubric for how many points equals how many hours. It always kind of ends up that way. And an interesting discussion I kind of want to try is if you sequence out all the things that you need to do and your estimate for not just how long you think they

would take on average, but how sure you are about that time. So in other words, there's some uncertainty on each, then what you could kind of roll up and get the minimum and maximum, right? So if something takes one week, plus or minus a day, and then something else takes two weeks, plus or minus five days, you could roll all those up and all those plus or minuses, and you get some absurd, I'll finish somewhere between one month and one year, right? Like you're going to, it's going to be not super useful. Or what you could do, which I saw someone saying is, is sort of play some Monte Carlo, you know, drawing from those distributions and actually show like, oh, the bulk, if you

assume that they're normally distributed or not. And then you drew each of these, you know, sequential things and did a stack up and you're randomly choosing and then looked at the outcomes. What you'll find is 95% of the time you're going to finish, you know, around six months plus or minus a week, right?

And then that way you can kind of communicate what the spread of that distribution kind of looks like, what the shape looks like, not just the extents. And then people can kind of internal what the spread of that distribution kind of looks like, what the shape looks like, not just the extent. And then people can kind of internalize like, oh, as it gets closer, you would expect it to narrow in and, you know, hey, that they landed, you know, two weeks out from what they said is actually perfectly reasonable. They did a good job, you know, I've never seen someone be willing to play this game, but it sounded interesting. Yeah, no, it makes sense. I mean, I think the high level thing is to definitely record these kind of interactions.

So if you're chronically over budgeting or under budgeting, you want to be able to adapt to that. You don't want to just like continue to sort of like recursively fall victim to like your own hallucinations. It's like you over budget and it'd be kind of forgot you over budgeted. And then, well, you know, nobody got fired. And then next half, like you double the over budget, right? So you want to actually have that accounting so you can say, oh, no, I actually thought we were going to get a lot more done this half. And I need to sort of like, you know, realize that and pivot for next time.

Another thing that's really important, you alluded to this with time is the dependencies, right? So if something's going to take a day, but you're waiting on another team, then it's going to take a day plus however long that team takes to do that thing that you're waiting on. And that could be a month, it could be a year, right? And so for time tracking, for project time tracking, it's good to use a Gantt chart, which a Gantt chart, if you haven't seen it, it's a way of representing sort of a graph of events. And so the events that depend

on other ones start when those events end and so the way the gantt works is you know anything that can be done right away just starts at time t equals zero and then when those things end the new things that are now possible start when those things end and so you can kind of see you know if if you had the way the gantt chart works is you know if you had effectively infinite head count and your goal is to just like go through these items, taking however long they take, this is how long the whole thing is going to take. And some things that might only take a little bit of time, they might be really tiny intervals on this Gantt chart, but they might be way out into the future because they depend on so many other things. So yeah, I mean, I think the timing part of project planning might be the most important because especially if other teams,

let's say another team, let's just say you have a project that you think is going to take three months and your whole project plan is six months, but it depends on something. And then that thing comes four months, you know, four months late. Well, now, you know, in advance, um, that you're going to be late. Like, you know, that you can't get that done in time because you weren't able to start at the right time.

Just kind of a meta point here, but, um, in my opinion, you always want to, um, announce your failures as soon as possible. Yes. That's my, yeah. Okay. You agree with me on this? Cause this is my opinion on it. I haven't actually asked anybody else's opinion on this yet, but, but yeah, I mean, if something is going to fail two months from now and you know it, you want to, I mean, not announce to the whole world, like write an email to, you know, everyone at company, Hey, I'm a failure. Like's not how it works. You want to tell the stakeholders, this, it's six months out, and I'm already telling you it's not going to happen,

and this is why. You want to do that quickly. And project planning, especially a living document, will give you that information when you need it. Yeah, I mean, I think there's a whole thing we could get into here that talks about like risk and the critical path. And we're like really running out of time.

So we have to zip through these things. But I mean, I think what you're saying is exactly right, Jason, that you can't know what parts you should decompose or de-risk or bring forward earlier, parallelize until you sort of see like what the transition of features you need to go through

that is going to take longest and therefore determine and also if you run ahead of schedule which seems to never happen then like yeah good luck else can move into the critical path that wasn't before or if something stretches longer you need to know when it moves from off to on the critical path at what time delay and so a Gantt chart is really in some ways underutilized and other words, overly prescriptive in some cases, but there is a balance there where it's a very useful tool. Yeah. I guess to wrap this up, a couple of things. One thing is don't get too caught up on tools. When I started, I mean, I was looking at the manager tools podcast and what I realized was the tools actually aren't that important.

Like you can use a spreadsheet, you can write it by hand in a notebook and the tools will help. But unless you're project planning, you know, all day as your job, you know, if you're spending, let's say five hours a week or 10 hours a week project planning,

the tools aren't going to shave that much time off of those five or 10 hours, right? So, the tools aren't going to shave that much time off of those five or 10 hours, right? So don't worry too much about the tools. It's really about building those processes and gaining that experience. One last bit about project planning is to be cognizant of the timescale

and recognize that all three of these scales I'm gonna mention are important for different reasons. So in the short term, so let's say, you know, two weeks or a month ahead, you want to have very clear kind of work packages. So, you know, people are being productive. You know, people aren't frustrated. If someone's overwhelmed that you know about that as well. And so that's, you know, we call it the agile would call those sprints. Right. But but effectively, you think of it as like a short term, you know, we call it, the agile would call those sprints, right? But

effectively you think of it as like the short term, you know, a couple of weeks to a month. You want to have in the medium term, let's say six months, you want to have milestones or maybe a year you want to have milestones. And you want to be able to have sort of a state of the union where you kind of reflect and say, okay, in the past six months, our team has done these things. And then the next six months, we're going to do these other things. And you want to really broadcast that and announce that and make that part of the ethos of your team. And so that's in the medium term. Now, the long term is looking out maybe three years, four years. And you might say, well, why do I even need a long term vision? And, you know, especially if you're doing, let's say, research or you're working on some or you're doing some project that's in maintenance mode.

Like, why do you need a three year, four year vision? And I think the answer is. Because ultimately, your project planning and your sort of culture will start to kind of blend together. And you're setting that long term vision will also kind of set the kind of culture and the kind of environment that you're going to have on your in your project or on your team. And so even if your long-term vision, it might be hard to say three years from now, you know, what research are we going to do? Like practically speaking, there's no way to know that you're really going to use that long-term vision to describe kind of your overall philosophy and sort of delusions of grandeur that you can create about the future where you want it to go. And so depending on how you structure that,

it will kind of guide the people who are interested in working on that project. I think a person put it to me in a really good way. It basically said, you know, someone's going to take a sort of big risk on something that has a big reward. And so that long-term vision is a way to sort of show that big reward to people so that they feel like they're jumping into something that has a lot of mass. So I think all three of them are super important. Again, don't worry too much about the tech, but we gave hopefully a lot of good tips here on how to do that effectively and how those things interplay. Nice. That was really fun.

Cool. All right, everyone. Thanks again for supporting the show on Patreon. We really appreciate it. And hope you all like the transcripts that we were able to add and the variety of other features that us, our producers have been putting together. Thanks again for listening and we'll catch y'all in a couple of weeks. Music by Eric Barndollar. Programming Throwdown is distributed under a Creative Commons Attribution Share Alike 2.0

license. You're free to share, copy, distribute, transmit the work, to remix, adapt the work, but you must provide an attribution to Patrick and I and share alike in kind.