Programming Throwdown - S1: Holiday 2022 Special

Episode Date: December 26, 2022

S1: Holiday 2022 SpecialToday we field questions from Programming Throwdown’s listeners about AI, machine learning, and more practical matters as developers in our annual holiday special!00:00:24 Introductions00:00:43 Programming Showdown merch00:02:13 Paul S00:03:28 Dealing with ergonomics00:10:39 On AI coding assistant tools00:16:43 Warren Y00:20:24 Ben inquires about performance testing00:27:39 Wild coding story00:29:37 AI coding’s disruption potential00:34:20 Jason’s Turing riddle00:35:50 ChatGPT00:43:59 Christian B00:45:13 Collection-of-Letters asks on documentation00:49:07 Zeh F00:50:51 Coding books that weren’t that great00:54:40 James K00:57:32 Jeremy S wonders about ML01:00:45 Virtual and live hangouts01:02:09 A retrospective01:07:49 Xu L01:09:22 Showing off the shirts01:11:31 FarewellsIf you’ve enjoyed this episode, you can listen to more on Programming Throwdown’s website: https://www.programmingthrowdown.com/Reach out to us via email: [e-mail] can also follow Programming Throwdown on Facebook | Apple Podcasts | Spotify | Player.FM  Join the discussion on our DiscordHelp support Programming Throwdown through our Patreon. Happy holidays from Programming Throwdown to everyone! ★ Support this podcast on Patreon ★

Hey everybody, this is our annual holiday episode. This is by far my favorite episode because it's a chance where we can chat with our listeners, people who we're ultimately serving and supporting. So really looking forward to this. We also have T-shirts to give away. And this year, we have the actual vendor, a new vendor for T-shirts. So it's going to be these really nice, high-quality vinyl print T-shirts, which I'm really, really excited about. I actually requested one of each design. And just because of shipping and, you know, I guess it being close to holidays and everything, I didn't get my t-shirts yet. So I'll make sure the t-shirts are good before I send it out to our winners. But several of our patrons will win t-shirts today, which is always a lot of fun.

It's always a nice show, I guess, to be able to give away free stuff. Yeah, totally. And I think, uh, it's also a,

you know, kind of good opportunity to answer questions. So we have a variety of different media here. We have questions coming in over email. We have questions coming in over the discord questions channel, which is the thing that I'm going to be checking most often. So if you're listening to this live, please post your questions over there. And we also have some questions coming in on Patreon, you know, DMs and all of that.

So really fun stuff. And we'll just kick it off with the first name in the list. As we always do, I'm not going to read out the full person's name i just don't know people have different um um you know tastes as far as privacy and everything yeah so so let's say paul s if if you're a patron uh and your last name starts with an s and your first name is paul you it's very very likely that you want a free t-shirt. So congratulations, Paul. And yeah, another nice thing about this vendor is they actually have a whole infrastructure in place for giveaways. And so it's much easier for people. People used to have to like play email tag with me to give me their address and everything. But I think we've got a new system

in place now where people will be able to just enter their address and get an order, a t-shirt, and the expenses have already been paid for you. So it's just like buying a t-shirt, except you don't have to. So you don't have to keep sending me your zip code and everything. So congratulations, Paul, on your t-shirt. Really exciting. That is very awesome. You want me to take it away with our first question? Yeah, let's do it. All right.

So I know there are some in here that Jason is very ramped up about. But I'm going to pick a different one. I'm going to pick one that's maybe a bit more off topic to start with. And there are kind of two related here, I think. I'm going to try to capture them both. And that is, how do you deal with ergonomics and muscle pain, presumably related to programming? And do you use special ergonomic equipment or have certain stretches? And related, and that's from, oh, I'm not going to attempt to say people's usernames. That's from someone in

the Discord channel. And then someone in the Discord channel. And then also in the Discord channel, someone was asking, which I think is somewhat related, how to effectively increase typing speed. So I'll take that away first. You know, I underestimated how nice it is to just have a really good adjustable office chair. And I will say that, like, I definitely had a lot of back pain when I was using essentially crappy chairs. And so getting a chair that has like a lot of adjustability, and then taking time to really fidget it into how I want it to be was a big help. Also, like as cheesy as it sounds, doing exercise, you know, specifically, like ab exercises helped me get rid of a lot of

back pain, which is the thing that was killing me the most about sitting in programming all day. And then for the like wrist and, you know, kind of like finger finger stuff related to kind of typing, I think being efficient about what you do, and that's a whole topic, and trying to limit sort of like keyboard to mouse movement is great. Also, I use a split keyboard, which helped me a lot, get it at an angle that worked a lot better for me. But I've never been great about doing stretches. I know that they recommend them in the courses I've had to take in various corporate environments, but I've never done those.

And I guess related to the kind of typing speed, I guess part of it is I'm actually not that fast of a typer. I think Jason's a lot faster typer than me, for instance, having observed him type before. But for me, it's about having a keyboard. The split one's really good for me. Making sure at one point I wasn't touch typing. I had to look because I was terrible. So making sure that i actually knew how to type and then just

making sure i'm efficient making sure that like i know what it is i'm going to type often that's the limiting thing not my typing speed so um uh if if there is an effective way to increase typing speed i don't know it and I've managed to do okay. I think like you can you can kind of work around that limit to how fast you type. Yeah, totally. Yeah. I mean, I think Patrick touched on a lot of the points that I would touch as well. I think I actually stand I don't have my desk goes up and down. I'm actually sitting now I sit when I do the episodes, but it's really the only thing I stand almost all the time. And so I don't have a good chair. I have some Ikea chair that I've had forever, but it's mostly just sits under my desk

and I'm just standing the whole time. I did have occasionally some back pain. And that was one thing that really surprised me was, you know, when I talked to a physical therapist, they said that actually you're kind of what Patrick said, you know, your core strength and your abdomen strength actually is causing you to overuse your back muscles. You know, like not having that core strength is causing you to overuse your back muscles. And so she suggested a bunch of core exercises like you can be on a pull-up bar and try and bring your legs up and all of these things. And that actually had a tremendous impact. And so it's one of these things that I would have never known on my own. I would have assumed if you have back pain, there's something wrong with your back. But actually the way that, that the solution that worked really well for me and, and yeah, I do try to, you know, do a variety of, of stretches and exercises maybe about once or twice a week. I also, I mean, this is a, this is pretty geeky, but I have a lot of fun with it. I have the Oculus VR, and I actually put on wrist weights and then play the VR games. It triggers that competitiveness. You don't want to get killed by the boss in a sword fighting game. But now you're wearing these two, three know, two, three pound weights on each

arm. So it makes it really hard. And if you can get through the Oculus video games with the weights on, it actually builds a lot of core strength. That's probably the best thing I've found as far as something that I want to do as a routine almost every day and get some exercise. As far as ergonomics, I have a vertical mouse, which I really enjoy. I've never really had wrist pain, so I couldn't tell you how effective it is, but I actually do enjoy just the way that a vertical mouse feels. It'd be terrible for video games or something, but when you're working, you really don't need to be that swift with the mouse. The vertical mouse just seems to feel a lot better. I don't have a particularly fancy keyboard or anything like that, so I I'll either bring a vertical mouse or I'll ask them to buy one and try and get some kind of standing situation.

And then what about typing speed? Do you think that's important? Do you have any tips for increasing your typing speed? got way better when I started playing those massively multiplayer text games. So it's like, you'd have to type like east to go east or I mean, some of them you've mapped to macros, but then you type like, you know, cast fireball and all this stuff. And they're actually like PVP. So it's like if you could, if you couldn't cast a fireball, then someone like killed you and took all your equipment. So my typing speed got way higher. But yeah, I think Patrick's totally right. You know, your typing speed probably doesn't matter that much. If you do want to improve it. I think there's, you know, a lot of these games, you know, a really good way to do that. At least for me, you know, I think,

you know, it brings it that kind of competitive spirit. But I wouldn't feel compelled to improve typing speed. You know, if your goal is to be a great engineer, you know, that is very, very loosely aligned with that. I mean, there's other things that you can do. And we've talked about a ton of them on previous shows that would be better for that. But if you do want to type faster, it's a good skill to have. I think a lot of these games are a good way to do it. Cool. So we'll jump to...

I'm actually just... I'll go top to bottom here. What are your thoughts on GPT-3 taking over? There's actually a bunch of questions along the lines of, what do we think about GitHub Copilot, GPT-3, and some of these AI coding assistant tools? I have to admit, I haven't used any of them, so I'm only going off of watching videos of

other folks. You know, I feel like if you're writing a lot of boilerplate code, then you're probably doing something wrong. So in other words, if you ask a mean if you ask chat if you ask a gpt3 to you know build a website for you it has no context of your code base so it's going to build you an entire website and uh it's not going to reuse anything um i think that's going to be really the Achilles heel here is, you know, I think a lot of these demos are done in a, like in a sandbox, where you say, you know, I'm starting your main.cc, right? And, you know, design, you know, systems like return radar energy, and I've seen some crazy videos. But then, yeah, the problem is, you have to put that in context of a much bigger system. And so you might ask Copilot to write something and it writes a ton of stuff, but then someone else could have done the whole thing in a function call because that's been implemented somewhere else. So, yeah, I would have to say I'm a little bit bearish on Copilot.

Of course, maybe that's my own bias is like uh not wanting to be put out of business or whatever but yeah i'm pretty bearish on it i mean i think there's other amazing uses and there's other questions about gpt that we'll talk about but i feel like writing code uh with gpt is maybe not the best use of it yeah Yeah, I guess on that aspect of it, I am not particularly keen. I do think, I would say more like the co-pilot sort of approach where let's just say it's like line suggestion

and like a better auto suggest as the default thing. And we've had a episode about that as well. I think those things can work. I think you can add a lot of, I don't know if this sounds bad, I'm going to be blasted for not being a general AI person. But I mean, I think you can add a lot of heuristics to some of that and make sure you limit it

to really high quality suggestions that fit, like you're saying, Jason, the context, right? So what is your next line going to be? You know, it's sort of like people for the first time getting really well done brace matching. It's just, it's really useful. And I totally think there will be some hinting systems that we see in the short term that work really well. But these long term where like, I just give a high level description, and it just works. And I've never seen a lot of thought to, you know, these questions say, like, hey, do this really large code base, and then I want to make a change, you know, or I want to maintain it or change it to do this or that. And like, do they just rewrite everything? Like, are they able to go in and make insertions? Maybe I just not look deep enough. But I think like, that's a big challenge. Like, you write your code,

you need to check it for correctness, but you also need to kind of know how it works. So you know how to fix it when it breaks. If you spend twice as long fixing that generated code as you would have writing it in the first place. It's not a super clear win to me. People are talking about in the chat, but I think that's something that I saw recently discussed and we were talking about it on my work team was like in a programming interview where you have a technical interview and they're asked a question. My joke was like your question's kind of bad if it can be completely solved in a correct manner by one of these bots right now at least uh and you can't tell that it wasn't done by the person

i feel like you're probably asking too much of a bog standard question and you need to ask something that has variation or exploration where the person needs to explain themselves i personally have zero concern about someone typing the words I said into one of these bots and then writing that code back and me thinking that they know what they're doing. Because that's like such a small part of how I do the interview, at least in mine. That's very much related to,

well, why did you choose that? Why not this other thing? Oh, you used a hash map there. Like, what is the, you know, like asymptotic runtime of that hash map? And then, of course, oh, it's constant. Well, is it? Like, what if you're, you know, here you're doing this insertion over and over and over again.

Are you really going to get a constant runtime if you're inserting millions of things into the hash map? Is it not going to grow? These kind of questions, right? And people who are relying on the chatbots to do that is not going to be effective. Yep. Yep. Totally agree. Yeah. You brought up an interesting point. We interviewed somebody and I know the new name of the company. It's now called SNYK, SNYK, but I forgot the old name um but we interviewed a company that was doing something like this right they were uh ingesting all of github and then using it to find errors in

your code and i think that's a great idea um i mean i think that has a ton of potential and so yeah to your point it's like these these kind like these kind of tools that operate at a lower level. Look for array out of bounds, look for not handling a return correctly, help you match braces. I think there's huge potential for things like that. I think it was called DeepCode, but I'm totally drawing a blank. But anyways, it's now called SNYK SNICK. So if you listened to the DeepCode episode and wondered what happened to that company, go check out SNICK.

And we'll jump into another winner. So we have Warren Y. Warren Y. So Warren Y wins a Rage Against the Machines T-shirt. I don't know if folks saw this, but we have two T-shirt designs designed by yours truly. I'm not going to claim to be as good a T-shirt designer as, I don't even know who designs t-shirts. Anyways, but I have the original programming throwdown design, and then I also created a Rage Against the Machines written with all of the fonts from the Fang companies. And we'll be giving away both, you know, some of those as well. So Warren Y gets a Rage Against the Machines shirt unless we get sued. It's not Bang anymore, though, because a lot of them changed names now, right?

Oh my god, mind blown. Wait, only Facebook, right? Did the other ones change names? Google is no longer Google. It's Alphabet now. Oh man. So yeah, it would be Mana or something. Also, with Netflix's Netflix market cap crash,

I'm not sure they belong in the FANG acronym anymore. Yeah, you're right. So, okay, if we had to redo FANG, it would be... No, we're not getting on this. No, we don't have to. It's not... No one takes us seriously on stock stuff. But I would say

these companies that are half a trillion market cap obviously stay. Actually, wait. Was Fang... Fang was Amazon, right? Or was it Apple? Apple and Amazon. There's two A's. F-A-A-N-G. What? I thought it was There's two A's. F-A-A-N-G. What?

I thought it was Fang, like Wolf Fang. I've always seen it as two A's. Okay. Okay. Well, so we have, we have Meta, we have Apple,

we have Amazon, and then we have Alphabet. So we have Ma, like a, it's now like a goat. Are there other companies that are really big tech companies?

Like they're half a trillion? This is a question for looking up on Alphabet's search website, Google. So yeah, I guess I'll have to change the shirt. For now, we'll have... Actually, the shirt is fine because the shirt doesn't say Fang or anything.

So you can tell people this is my Rage Against the Machine shirt where I have all the fonts from Ma. Well, dude, you have like perfectly picked it so that Tesla falls ever so slightly short, 476 billion. So it's only Amazon, Alphabet, Microsoft, and Apple. Wow, Tesla was $800 at the start of the year, weren't they? Yeah, something very high. Oh my God.

I mean, that is wild. I mean, I guess I think Facebook meta fell further than Tesla though. Meta is only $318 billion now. What? Wait, really? Wait, so Meta is not even in it according to my last definition. Tencent and TSMC and NVIDIA

are all higher than Meta and Tesla. Oh, wow. Okay. So it's really just, when I look at the economy in the end of 2022, all I can say is, ah. okay there was one on here good oh yeah here it is this is from Ben he says I haven't heard a lot of discussion around performance testing what do we think about that where does it fit in the software development life cycle kinds of nasty problems can you find is it worth it and is this a skill that a software an average software a typical software engineer should have or defer to experts. Um, this is one of those, uh, right.

Like cliche phrase for it, but, um, it's one of those things where everyone loves to just go early optimization is the root of all evil don't don't optimize your code. Uh, I think the like real world answer is it's more nuanced than that. And I think, to me, if you're going to perform optimization or you're going to try to write a vision code and you don't have some kind of performance testing in mind, I don't know. I want to make strong comments,

but I'll refrain from saying that it's not worth it. I'll say that you should generally have an idea on how you would measure performance if you're going to try to optimize something or improve something. And I will say that I, in my coding, try to strike a reasonable balance. If the optimization is going to be very, very difficult and involved and make the code hard to read, and I'm not sure it's going to work. And you're almost always better off not doing it. But that being said, if you are building something from the outset that you can tell is going to break, not at 10 million users, as an example, but at 100 users or 1000 users, that's kind of a problem, right?

Like, if you have a reasonable expectation, you're going to get more than that. And so I've seen people build something that fell over well within the sort of first expected amount of work it would need to do. Then they rewrote it to barely pass that only for it to fail over again when you had a 50% increase in the throughput needs. And so I think you can get

into this problem where you're in a loop of barely making it work for the load you have, barely improving, barely improving, and you end up just really trying to incrementally improve something that needs to be kind of thought again from the ground up. But on the other hand, I have also seen people take tons and tons of time to build a very beautiful, well-optimized system that never got finished or never saw the light of day because they just took too long or misjudged. So you got to have some balance in there. But I will say that I do think every, in my experience, every engineer at some point, software engineer needs to know how to approach timing and measuring the performance of some code,

what the kind of underlying system impacts are into the measures of those performance. And I've worked with people that went and did what they thought was performance measurement and came back and effectively all they were measuring was noise. There was the signal they were looking for. They weren't testing it in a way that was going to give them the answer they want. And I think it's something that you don't have to be understanding of all parts of the system and, you know, amazing. That's your one and only special skill.

There are people like that. But I do think everyone needs some minimum competency in being able to assess how their code runtime looks and the performance of it. Yeah, that makes a lot of sense. So Ben, I have experience with performance testing. Performance testing is a type of testing that is conducted to evaluate the performance of a system or application under a given workload. And then the rest of the chat.

And then the rest of the chat. So yeah, Patrick is definitely the key person. I don't have a lot of experience with this. I will tell you a really funny horror story, though. People assume when they come to Google that all the code they see is going to be amazing and everyone's going to know how to write in every language and all of that. Not true. I mean, it's just normal people, right? And so I was in the AI org, and we had to stand up a website for like a dashboard to show the results of the model.

So people who are using our model would have something to look at to see if it's getting better or worse. And the dashboard took maybe like two minutes when you hit the refresh button. I mean, maybe like anywhere from 60 to 120 seconds for the dashboard to go from, you know, I type in a URL to I see the result. And, you know, we just figured it was, you know, we were just pulling a lot of data. Like when it pulled up, there was tons of data and tables and everything. And so natural thing was like, oh, there's all this HTML, and we have to go and fetch all this data. So finally, I said, okay, we had a fix-it week.

For people who don't know what that is, that's a week where you pause all of your objectives, and you just focus on making the code better. And no one wanted to touch this experiment dashboard. People were doing things around it to make it draw things without having to touch the experiment dashboard. And I finally said, okay, I'm going to rewrite the experiment dashboard in a week or at least get as close as I can and then keep working on it. And so the way the experiment dashboard worked, there was a function call, which is like HTTP get, but it's this was all in, in, in I think, Java or something.

So there's this one function call that returned a bunch of HTML. And the way it did it was, it had a string object. And so it was like string s equals, you know, open bracket HTML. And the next line was like, s plus equals, you know, like heading, and the next line was x plus s plus equals table. And then like, you can imagine the very last line of this function was s plus equals, you know, close bracket HTML. And in Java, every time you do that plus equals, Java creates like another copy of the string, right? That's big enough to hold both of them. And so, yeah.

So literally I just changed string to string buffer. That was it. And it went from taking 120 seconds to taking like 30 milliseconds or something. That's actually impressive. Yeah, you just have to add the word buffer. And then at the end, you have to like convert it back to a string. It's like, oh my God.

That's where GPT-3 could come in. Like it could look at that function maybe and say... And totally replace that that individual low bar though gpt3 looks at the function and then fires people oh dear i i heard we'll pivot to a crazy story i don't know if you can confirm or deny this but i heard from people at twitter that the original plan was to print out your code on paper and then Elon and other folks would look at the stack size of the paper and then use that as like one of the heuristics to decide like who stayed and who left. So you show up with, you know, a dictionary,

you know, size stack of paper, and you get to stay. That's what I heard. Yeah, I guess my fussing at the team to stop directly importing the generated proto-buff auto-generated code and having it make it from the proto-definitions instead, I guess, would hurt my stack.

I should have just left all the auto-generated code in. Yeah. Can you imagine? So, you know, you imagine this like line of people and this person is, you know, walking down the line and all of a sudden this one person has a six foot high stack of paper and it's like, yeah, I wasted, you know, six feet of paper. You're fired anyways. That's too much.

It like overflowed, the int overflowed and now you're at negative. Oh my gosh. I mean, let me just say, you know know i feel like compelled to just put a disclaimer i mean it's terrible when what's happening at twitter and folks getting laid off and everything so i mean what but i feel like it's you know you have to be able to make fun of of everything so so uh um yeah so yeah i do feel for those folks about trying to trivialize it, but a good joke is a good joke. All right. So we have, let's see, going down the list here. Someone asked, what kind of disruption to Google and Stack Overflow will ChatGPT have

to the search industry? That's an interesting question. Have you thought about that, Patrick? It has huge implications. Ooh, does it fall? Does this just mean like in general, I don't know when to say like spam, like people putting up wrong answers

or overwhelming creating like fake content pages. Is that what they're asking? Are they meaning like doing something better? Rather than giving you a website result in Google, they would on-demand generate it? Well, for example, let's say you go on Stack Overflow and you ask a question like, how do I sort a list in Python or something?

So someone who wants to stack overflow, you know, virtual points or whatever, we'll just put that in a chat GPT, you know, post the answer. And then, you know, if they're right more than they're wrong, they end up with like positive karma. And they could just basically have this infinite karma generator yeah i mean i think like anything this is just a general like bad review problem like what i guess my question there would be what do do we eventually need to share the economic costs of running chat gpt like it costs money to run it if everyone's just doing that and it's very expensive to run it becomes like not that much different than you know hire outsourcing people to do the same thing so it's this like not to make it about economics like what is the reason for someone to do that like can you change the economics such that it's there's like no reason

for them to do that um it is unfortunate like to have the deluge of stuff people could just write bad answers today i've thought a ton about it i i feel like it will be challenging but i i feel like there will be solutions to it i feel like there will be ways of asking you to change something or i don't know some challenge response that you could do to some of that to kind of like weed out. But maybe they'll also, it'll be a cat and mouse game, I guess, but maybe it always has been. Yeah, I wonder, you know, talking about the economics of it, obviously training the model requires an extraordinary amount of money. I saw

the figure, I feel like I'm going to misquote it, but it was an absolutely extraordinary amount of money. But I actually don't know what it costs to serve the model. I'm not sure if it is that expensive. I really don't know. I know that OpenAI will give you a pretty generous number of queries for free, but who knows what they're paying for those i guess the other thing i said though like if if they're so good that they're close or indistinguishable like does that not also mean maybe the companies can turn to using them themselves like i was sort of saying like generating you answers you want on demand and then they capture that instead of allowing other people to do that and then they don't even Like I was sort of saying, like generating you answers you want on demand.

And then they capture that instead of allowing other people to do that. And then they don't even need that whole problem, right? Stack Overflow is only useful if there are people on the site answering questions. But if you have a system that can actually reasonably answer all those questions and you don't need those people, and then you don't have this like, it's not exactly the marketplace problem, but you don't need to bring question askers and question givers together. You can stand up a website that's like a better Stack Overflow because it answers your specific version of the question the first time, every time. Yeah, really good question. I mean, really good point. Yeah, you know, it's amazing. That's a phenomenal point.

I imagine that there'll be almost like a Stack Overflow Visual Studio Code extension where if punch in like, how do I center a div? And then chat GPT tells you. And then if that answer doesn't make sense or is wrong or you just can't interpret it, then you go to Stack Overflow. So it kind of like eliminates the whole bottom layer of questions there. I actually, I had a riddle that i remember from grade school

and i asked chat gpt3 and it got it right um so so i'll ask you patrick here turing test is patrick better than gpt3 so solve this riddle when you put this in a bucket, the bucket gets lighter. It's not, is this word for like emptiness? When you put this in a hole? Yeah, that's right. All right, it's official. Chat GPT-3, not as smart as humans. Hashtag solved.

Same humanity. This is not an AlphaGo scenario yet. I'm so glad. Oh man, I'm so glad. Oh, man. I'm surprised, ChadGPT, that you got that right. It's like literally, you know, it says the answer to the riddle is a hole and then it explains

a whole bunch of stuff about putting holes in buckets. It'll be interesting, you know, a lot of maybe bad has come out of the internet, but the amount of like machine learning input to be able to get to this point by basically just feeding it everything that's on the internet is crazy to me. Yeah, exactly. One second here. There we go.

Cool. So, yeah, let's see. We have a lot more questions about chat GPT3 or chat GPT that we could opine on. Actually, so, you know, maybe I'll just kind of briefly explain what chat GPT is for the audience, because this is something that I think I know pretty well. So, so GPT-3 is a forward model. So the idea is, you know, you can go on this Discord, and there's a whole bunch of folks writing sentences. And you can train a model that says given the sentence you know the fragment of the sentence up until now so maybe take the first five words the first seven words

of the sentence you know predict the eighth word or maybe say you know given the first five words predict the next five words right sometimes it's it's, you know, almost impossible, you know, because it's like, you know, I went to the book, you could have gone to anything, but many times it actually is very possible, right? As many cases where, you know, you almost know what the next word's going to be before the person says it. And so, you know, it generates a probabilistic model that kind of guesses the next word. So that's GPT-3. And given enough data and the right kinds of data, a lot of curation, all that, you can do this.

And so then you could just say, well, here's like a prompt, or here's the start of a sentence, just keep generating the next word, and then use that as the input again to generate the next word, and then use that as the input, again, to generate the next word, and then do this again and again. What chat GPT does, the chat part is where what they did is they had GPT-3 generate several paragraphs for the same query. And then they had humans go in and pick the best one, or even just rank all of them. And so now you have a supervised signal. You have a person saying,

okay, here's two sentences. Here's two paragraphs. This one's better than that one. And so you could do what's called contrastive learning. It was also called learning to rank back in the day, but now it's called contrastive learning. And so using contrastive learning, you can fine tune GPT to produce things that would be ranked higher according to people. That's the chat part. And yeah, that adding that piece is really been extraordinary. You can tell there's a perceptible difference between the two. I think it's yeah, like that fundamental way of trying to generate the negative. I guess it's really interesting. How much of the problem is finding the framework or the framing of how to sort of like set up things so that they work that way?

And how much of it is just like sizing or technology in the sort of like machine learning training itself? Yeah, it's a good question. I mean, I think that the techniques here are not that advanced. I think the hard part of GPT and chat GPT, there's several pillars here. One is it's really difficult to do this at scale, right? The scale, the cost, and all the complexity of that, and being able to iterate. I mean, GPT-3 is not literally the third iteration of GPT.

I mean, they've done thousands and thousands of tests, right? And so being able to do that at scale is hard. The data curation is really hard. I mean, I think that's the secret sauce. I think a lot of people take for granted, they think that, I'll just download Wikipedia. And if I had enough time and energy and electrons, I could just train it. But I think they did an extraordinary amount of curation. I think there's all sorts of linguistic curation going on before the training starts. And then now you have all of the human labeling

that they've done to do that last step that I referred to. That's also extremely expensive so um um so yeah i think that that that's actually where all the complexity is here i mean there's other things like alpha go where it's complex in different ways but here actually the algorithm i think is relatively straightforward and what do you think about the like you see it headlined everywhere like jailbreak or which i didn't understand what they were saying at first. Like I did, I had to kind of like look and like, oh, okay, I know what they're saying. Like basically people putting in guardrails or safeguards, like, oh, if we're doing image generation, we don't want real humans or we don't want nudity or we don't want people talking about racist themes. And so trying to put in what amounts to like, the model itself knows how to do those things. And so you need to kind of tell it not to at the end.

Like, what are your thoughts about that? Like, is that just a losing battle for them to kind of approach it that way? Yeah, it's a good question. So the thing about chat GPT is they, you know, they have that really powerful supervised layer at the top. And one thing that's really important is the top layer, the layer that's closest to the output, has a massively outsized impact. So, for example, look at MKL, which is...

Oh, I forgot what it stands for. I think it's a multi-kernel layer. But anyways, it's basically MKL is where you have a deep network, right? You have some neural network, but the very end, the last layer, you use what's called a Gaussian process, gaussian process which is a very computationally expensive uh process um um so you couldn't you couldn't do it in the whole network but you just do it at the end and that by itself is really powerful it's able to affect because it's back propagation right doing something at the end you're able to back propagate your signal all the way to to back. So, um, so yeah, I mean, if you look at chat GBT by putting the supervised signal at the end, chat GBT ends up being very, very good at,

um, sort of like dodging questions that it doesn't want to answer saying certain things are, are, are, um, you know, um, like not okay to talk about and kind of like always trying to like steer the conversation in certain directions it has almost like an uncannily human like politician political politician like ability to be able to do that now that's all the supervision um and yeah i think what they showed is that, I mean, you can actually, I've seen, you know, Twitter posts and stuff of list of prompts and then paragraphs to talk about and then rated it. So, for example, someone might say, the person who's building ChatGBG3 might have a prompt that says some like racially insensitive question. And then there's the three generated answers. And then they manually put in a fourth generated answer, like that they write by hand that says, you know,

I don't want to talk about this. It's offensive. And then they assign that one, like a huge positive score and like give a huge negative score to the others. Right. That's basically what's going on here. And that works, but that's going to have some amount of coverage. There's going to be, you're not going to be able to conceive of every single way of presenting, you know, that content. And so, you know, yeah, people always, you know, find a way to sort of like punch through that, I think. And so actually I asked ChatGPT3 to announce our next winner. And so we'll see what he said.

So Christian B had been wanting to add a new T-shirt to his collection for weeks. I was excited when he saw the programming throwdown one at the mall. At the mall. Okay, well, let's not get ahead of ourselves. Christian B. stopped by the store to see if it was in stock and was pleasantly surprised when the salesperson offered him the chance to win a shirt by answering a trivia question about surfing. Christian B. confidently answered that the question was Kelly Slater and was thrilled when he was correct.

He left the store feeling grateful and proud to add the T-shirt to his collection. So, Christian B., you got a T-shirt and a story from ChadGBT3. Congratulations. Oh, dude. be you got a t-shirt and a story from chat gpt3 congratulations oh dude i sense a i sense a new future for all of these things where yes we get wordplay from our overlords kelly slater is uh that was uh what was that saved by the bell right yes oh man i just we just dated ourselves all right i got a couple of questions that i think a little bit related here so first one from a collection of letters as a username is it better to refer to documentation for a beginner or follow a video guided course And then for a young wannabe programmer, what should be the key focus points or checkpoint?

I think this is a great question. I generally don't tell people to start with reading the documentation. I guess it depends if you are looking for a specific answer to something or if you're trying to learn how to do a new thing. So just making up an example, if you want to learn how to use Unity or Unreal Engine and you go straight to the actual technical documents, probably not going to help you if you're not already

a video game programmer. Maybe it'll help some people. I'm not saying it can't be done. Just probably a bit difficult. Whether that you go to a video course, or just a general tutorial, those are probably going to be be more helpful. That being said, the more niche you get into a topic, if you're in some specific, you know, shared pointer issue and deep in the depths of C++ with templates and something,

probably going to be difficult to find a whole video guided course or even a tutorial on it. You're probably going to get something much more akin to technical documentation or maybe a question on Stack Overflow, something like that. So there you're definitely going to have to be. For someone learning to program, the key focus points I feel like are driven from an interest. I think if you can try to do something that keeps you engaged and interested, I don't think it's Malcolm Gladwell was exactly right with whatever it was, 10,000 hours to become an expert at something. But I do think you just need time doing programming and programming adjacent things for it to click. And so whatever can keep you motivated, I feel is going to be better than trying to hit some specific.

And I think that can be what makes it difficult that there's not you just do A, then B, then C, then D, and then you're a programmer. I think finding a way for you to try simple problems to make progress, but always be working. I do think, I guess, trying to finish things or at least get enough progress on things and so that you're doing iterations and doing pieces of the arc, the software development lifecycle, however you want to call it, that you're doing it and getting repetitions on it. I think that is useful. So sitting down and saying, I want to build, you know, Dwarf Fortress, and I'm going to just start from scratch. That's pretty difficult because it's gonna be a long time before you kind of circle back

in a meaningful way having completed it. But yeah, I think mostly trying to align it to your sort of like interest and desire will keep you motivated and keep you spending time in the subject yep yep yep totally agree um yeah i think i think yeah that's a really good point i mean i think you should always have something kind of cool that you want to build because that that's also a sign to you that you're kind of entering the right discipline right if you like if you kind of follow a bunch of tutorials then you know you might build some skills but if you go at it from first principles and say i want you know to not have to compute my tips by myself i want my app to do it um then then uh um you know if you kind of exercising you know the right muscles to turn that into something real and then getting that feeling at the end that that's gonna kind of

tell you a lot more about your connection to the discipline cool so yeah i have an email question that i'll that we'll get to but first let's announce another winner since we're running a little bit low on time um so i hopefully i can say this right so the first name is z e h and the last name starts with an f so z i'm assuming that's z um you won a rage against the machine t-shirt and uh i asked uh chPT3, it said, Zed, a skilled hacker, learned about a contest offering a free t-shirt to anyone who could successfully hack into a clothing company's website and retrieve a hidden message. After several hours, Zed found a way in and located the message, sending it back to the company as proof of his hack. The company thanked him and offered him the free t-shirt as promised, which Zay eagerly accepted and wore with pride. Congratulations, Zay, on your t-shirt.

I look forward to the future when people get their little narrative stories on the t-shirt that they win. Oh, man. Okay, next year, we got it. That's actually a great idea you know connect chat gpt to like cafe press or something i mean we get a lot of christmas cards i guess maybe this is an old person thing but like you get a lot of christmas cards from people you've met over the years from family whatever and they always include a little story i think i'm just to get chat GPT to write my story next year. Just like, I have these kids. They generally were good.

They were in school. And I have a wife. Like, make a letter. That'd be amazing if they could lay out the photos and everything, too. Oh. That'll just generate the photos. They don't need to be real. That doesn't have to be our family. I mean, who cares?

Related to the previous question, this one was good. And then maybe we could do another email one. Coding books that were great, but time proved them wrong. That's an awesome question. I tried really hard to think about one here in the last few minutes. I didn't come up with any. There have to be some pretty funny ones from 20 years ago, 30 years ago, right? Like, oh, we have to be careful because computers only have kilobytes

of memory and something, right? Just all the classic misunderstanding the exponential growth. And then on the flip side, I bet more recently, you probably find some giving you guidance sort of like, yeah, don't worry, the computer will always get faster. That turns out not to be true. That would be like an easy one. I'm trying to think of there are probably like dead methodologies as well, or like dead programming languages or languages that didn't really go anywhere that people thought were going to be really hot for a while. Anything Google does. I think like think you could just probably find books about those things that are no longer useful. But Jason, you have any books

you read and you're just like, this is just bad advice? Yeah, I mean, I'll have to check. I keep all of the notes on the books I read in this website that I built the front end using Dart and the back end is using Golang. And actually, Golang is popular. But yeah, I would say, OK, there's two angles to this. One is there's been several AI winters, right? So folks have promised the whole world and then turns out, oh, AI is actually not that easy. So that's definitely there, but yeah, as far as technology, I think that, uh, um, I mean, there's definitely a whole bunch of defunct, uh, or, or just like libraries that are just, uh, languishing, right. I mean, dart comes to mind, but there's a ton of

others. So yeah, there's, I wouldn't, I don't know if it's a thing about books or books, especially books about programming are usually not very opinionated. It starts to get really opinionated when you talk about AI or the singularity or these other things. But yeah, I mean, I would say there's probably been just like huge volumes about how to, how to code in Dart or how to write in some language that never really took off. I remember reading day trading books when I was younger. I don't want to say that this was before the dot-com crash,

when everyone was sort of day trading. I remember going to the library and seeing books about day trading, whatever, that just a few short years later would have turned out to make you like absolutely bankrupt. You know, do you remember, like, you know, when you have these market crashes or, you know, in the hype phase, like at peak hype, there's always those people who are day trading and telling you how amazing they're doing and everything. You know, maybe that's like, there's got to be someone who's looked into this and said, oh, and when everyone starts telling you how easy it is to day trade, that's a sign that's all going

to fall apart. We had this calculus professor in high school who was literally day trading. He would give a lecture and then he would give us some worksheets and then he would day trade and uh on the computer in the classroom and that was what 1998 97 or something so then like yeah the economy fell apart a year later i thought you were going to go crypto on that one people who talk about crypto all the time and then just disappear um i have a interesting kind of meta thing here i asked chat gbt3 to write a story about a coder and i'm not going to say the name yet because i'll spoil it but who gets a free t-shirt by dividing by zero and and chat gpt3 like uh chat gpt literally can't do it like i'm getting all sorts of weird errors and everything like maybe putting divide by zero

in the question is causing gpt to have a heart attack like it's just having a really hard time it It's just seg faulting. Oh dear. Yeah, I asked chat GPT to tell me when it halted and OpenAI just went out of business. Just ask it the halting problem and then just somewhere there's a fire. Oh yeah, one thing that really confuses it that I get a huge kick out of is when you talk past the sale

you know so when you say something like explain why the halting problem can be solved if you use haskell so it's like you've presupposed something that's totally bogus and it tries its best i mean it's uh it's so so but but yeah it generates some absolutely hilarious things oh it won't point out that you're like you're giving it something contradictory right right i mean sometimes it does but the majority of time when you talk past the sale it will just try its best to make that a reality oh here we go i got it okay it's up but the fourth attempt it actually tells you how many attempts um i don't know why i had to take four for this but um james k james k our next winner was a skilled coder who loved nothing more than solving complex problems one day while deep in thought

james had an idea he remembered reading about a strange phenomena known as a division by zero error. If he could trigger this error, he might be able to find a way to win a t-shirt. Excited by this, he typed out one line of code that would trigger the error. To surprise, the code worked perfectly, and he was able to get a solution to the divide by zero problem. As a reward for his hard work, the company he worked for offered him a free t-shirt. James accepted the prize with a smile. The end. It actually said the end.

We need to generate stories about two handsome podcast hosts becoming enormously wealthy and out of their generosity giving out because they gave out free t-shirts. There we go. Oh, that's a great idea. Write a story about two people who gave out free t-shirts and somehow tripped and fell into a wad of cash. Oh, man. Yeah. Oh, yeah.

Let me get to the email question. I definitely want to cover that. We have a couple here, but let's do the first one. Machine learning is, I'm going to summarize here, and this is from Jeremy S. Machine learning is a really interesting topic to me. I want to jump into that field, but I'm not sure where to start. Yeah, I get this question all the time um so you know sorry if this sounds like i can't answer but i've thought a lot about this and and so my take on this is is the following um at the end of the day like if you are working on software you know you're building some type

of software product for some audience right and so and so when you build it you try and use common sense, you say well I would want it to do this or I want the compiler to tell me quickly when there's an error or I want the map when I open the maps app to start at my location, so that's all common sense but sometimes you get it right and sometimes you don't to start at my location. So that's all common sense. But sometimes you get it right and sometimes you don't. And the only way you really know if you are thinking on the same level as your audience is to collect data, collect metrics, and try different things and see how that affects

the metrics, right? And so you can't actually, you know, if you work for, like, you can't actually build the best product unless you're measuring things and then reacting to those measurements, whether it's a game or an app or a website or anything. Even a compiler, they need to ask, you know, their audience, you know audience questions and learn more about their use cases. And so you can't build software without running headfirst into this problem. And you can't solve it without some amount of statistics and machine learning, right? I mean, even if it's as simple as I'm going to, you know, group of my bug reports and work on the one with the most, the topic with the most reports. So that's still some way of looking at metrics and then reacting to them. And so you could argue,

well, how did I do that grouping? Did I do it by hand? Can I automate that? And so this is a really good place for people to start. So if you're a software engineer, you want to get into machine learning, a really good conduit is to find the team that's collecting metrics and learning about the product and just embed yourself in that team. And you invariably hit like all of the most interesting problems in machine learning. You're not going to hit, you know, the things like computer graphics and stuff, but all the interesting problems related to sort of decision theory and all that you'll hit with almost any software product.

That seems pretty reasonable. Yeah, I don't have any specific advice on that. But yeah, I mean, I think, yeah, it makes a lot of sense. Yeah. Sorry, I lost my place here. Patrick, do you have any questions you want to cover? Are there any from the chat? I think we got most of them from the chat.

I didn't curate the list of the email ones, but I'm looking here. People are wanting a hangout, Jason. They want a way to meet. I think we can virtual one easy and person one. I guess it depends on where. Let's see. Looking for any

others. I think we got a lot of these covered. Yeah. I'm terrible at email. I apologize if i lost your question just uh either uh post in the chat and the questions channel on discord and we'll make it up to you but uh yeah for some reason i thought i had a second one actually it's gone somewhere else but uh um but yeah this was awesome i mean we covered a lot of good content let me do let's do the final uh sixth t-shirt so um what should we have this person do patrick to get their t-shirt uh is this the last one this is the last one? This is the last one. Have them give an amazing outro to a podcast.

Oh, that is a great idea. We're going to make it very unhappy. There we go. All right, well, ChatGVT is kind of thinking about that. Maybe we should do some predictions. Oh, actually, so last year, if I remember correctly, I wonder if we have it in our notes. But last year, I remember one prediction that you made that really stood out was that you felt like space was going to be kind of like a big motif for the year. And I think you kind of got that right um you know i think that like i

personally got really into spacex and and all that i mean i got in you know really into the hype there yeah there have been like a crazy number of launches this year like i mean i think i was hoping starship but they have some regulatory. There have been a crazy number of launches. The Spin Launch team had a pretty cool test. A number of other projects have marched forward. I think it's one of those pretty awesome things that continues to move. I don't think we're there yet. Look at the Starlink became kind of a big thing this year, right?

And we saw a bit with stuff that happened in Ukraine. And people started to see that this is a pretty transformative thing. And this is like already up and running. This is like not a one time going to be there. It's there now. Yeah, totally. I mean, one thing I said, I know, I'm cherry picking things that that had the most consequence or were most correct on so you know i totally understand but but one thing that i talked about a lot last year for predictions for next year was uh that education was going to just like radically change and um i actually predicted that they would privatize scholarships i mean i was you know for for and I don't know if my prediction was a year or not, but, but, but looking at what happened there, I mean, a number of things did happen along those lines. So one, you know, college enrollment has completely tanked. So,

so, so people are just not going to college at the same rate. So that's a seismic event. There was a whole effort around student loan forgiveness that I think it fell through at some point, but it was in the zeitgeist for a while. Oh, that's right. Yeah, the court disputed it. Yeah. But it was in the zeitgeist for a while. And it does seem like, I do think that my prediction is going to come true eventually that that you know there's going to be some type of privatization of higher education like to a much greater degree it's gonna be like a seismic shift towards privatization um so yeah i think we're marching towards something there are you gonna make new ones i think i'm gonna be boring and just stick with like space being like a huge thing this next year. But maybe you have some more interesting take.

Yeah, I think that I think that the same predictions are just continuing to play out. Oh, okay. I do think that I mean, this is not anything mind blowing, but I do think there'll be some type of economic collapse in 2023. I mean, everyone's kind of saying that. But what's your take on that? I mean, do you go with the flow on that? Or do you think it's people are being too bearish? I mean, look, everyone's saying, oh, they knew, they knew. People at work are saying they knew. I mean, to be fair, people have been saying there would be something like that

or this or however long. I heard some discussion with, I don't know the name now of the person, but it was an economist. And they were saying what they think, and I was a good take on it. If you look at times that were most similar to now with this steady margin inflation, Fed raising the rate in the United States, stuff we see around the globe. This has kind of happened to us before.

And it turns out it's just really boring. You just end up with a really slow slide down and then a really slow recovery that we're used to in the last 30 years, like bubbles and pops and crashes and then V-shaped recoveries or things at least happening one side or the other very, very quickly. But I think their take was like, this is something that will probably play out over like a decade. So when you have like a U-shaped thing that has like, it does have a minima, but you don't feel it so strongly when it's U-shaped

and not V-shaped, right? And so this may play out over the next, you know, sliding down the next two, three, four years, you'll start to get mixed things. Some things will start to stabilize, some things will get worse. Then you'll start to see a few things getting better, less things getting worse, more things stable. And then eventually things will tip back up and we'll be in a recovery. But you won't be able to know it, you won't know it till later. And I feel like that's probably a reasonable guess about what's going to happen. Yeah, that makes sense. I mean, thinking back to 2008, 2008, there was definitely like a crash that spanned over months, right? I mean, I'm trying to struggling to maybe even less than that and then you're right the recovery really took maybe what four or five years or so i do remember there was a period maybe it's like 2010 or 2011 where i just felt like

the economy has been bad for so long it's like you almost i started to forget what it was like to have a good economy. All right. So we asked Zhu. Zhu is our last winner. Congratulations. It's X-U. So I'm sorry if I'm butchering that, but I believe it's Zhu. Zhu L is our last winner. And we asked him to, we asked ChatGVT to write an outro of the podcast. So here we go. Zhu is a big fan of the podcast Programming Throwdown,

and he loved listening to the latest episodes when they were released. One day while listening to the latest episode, Zhu had an idea for an outro that he thought would leave a lasting impression on the listeners. And here's Zhu's outro. Thank you for tuning... Yeah. I'm literally reading this on the fly, so help us here.

Thank you for tuning in to this episode of Programming Throwdown. We hope you enjoyed the discussion and found it helpful. If you have any comments or suggestions for future episodes, please don't hesitate to reach out to us. We always welcome feedback from our listeners. Thanks for listening. As a token of appreciation, the hosts offered you a free t-shirt with the show's logo on it, which is actually exactly what we're doing, so thank you so much. It was amazing, but it was so vanilla.

It was good, though. The only part of this we can't validate is that these folks are wearing them with pride. I notice every answer from ChatGBT3, every paragraph ends with the person wearing the shirt with pride. It's up to you. take photos of yourself wearing your programming third-hand t-shirts with pride and add us on Twitter. I guess actually, you know, just to sort of close the whole loop here is

I do think that these generative AI models are going to be a huge, huge part of the next three, four years. Because, I i mean just imagine all the different things that you're going to see disrupted like spam is going to look completely different next year i mean that's yeah that's the i don't know why i keep going to that such a negative example but just you know i mean i mean like all the like okay imagine if you were what's that company that has the images you know like if you need an image for a presentation, you could pay them five bucks.

Shutterstock, I think. Yeah. Yeah. Yeah. Like Shutterstock is going to be, you know, they're under enormous pressure now, right? Because you could just generate a photo and who cares if the person, you know, doesn't look exactly right. So, yeah, I think it's going to be just extraordinary what's going to happen oh the other part of this is people are generating pictures and texts and it always starts with like very low level sensory things like pictures and

texts but imagine if you could generate games or you could generate um like imagine if you could generate games or you could generate, like imagine if you could, if your goal is to train a, like a police officer to decide who is at fault in like a traffic jam. And so you could like generate like more and more like confusing traffic jams and like, you know, it could get harder and harder and you could train them. Or imagine if you could train them or imagine if you could generate the advertising so like instead of having to find the picture and the text for the

ad i mean i guess it's that's the same picture and text stuff but but i think the generative stuff will become sort of domain specific and uh that's where it can get extremely extremely interesting i i dread our future. It's going to be amazing. All right, everybody. Hey, this was a pleasure having so many folks here with us and, you know, folks in the chat and over email and supporting us on Patreon. Thank you so much for a year of that.

We really appreciate all of your time, your energy, your feedback, and your years. I appreciate having those for an hour or so every couple of weeks. So thank you so much. And I'm looking forward to an exciting 2023. We have another lineup of really exciting guests. Patrick and I are also going to do a few, you know, a bunch of duo episodes. So that's going to be a lot, you know, a bunch of duo episodes. So that's going to be a lot of fun. And we're looking forward to all of that. Music by Eric Barneller. Programming Throwdown is distributed under a Creative Commons Attribution Sharealike 2.0 license.

You're free to share, copy, distribute, transmit the work, to remix, adapt the work, but you must provide attribution to Patrick and I and sharealike in kind.