Programming Throwdown - 165: Differential Equations

Episode Date: September 25, 2023

Intro topic: Revisiting the power of SpreadsheetsNews/Links:LK-99 Isn’t a Superconductorhttps://www.nature.com/articles/d41586-023-02585-7Normalizing Flowshttps://pyro.ai/examples/normalizing_flows_i.htmlHow is llama.cpp possible?https://finbarr.ca/how-is-llama-cpp-possible/Chat with open source large language modelshttps://chat.lmsys.org/Book of the ShowPatrick: Math with Bad Drawings by Ben Orlinhttps://amzn.to/44dsgDzJason: Overboard! https://play.google.com/store/apps/details?id=com.InkleLtd.Overboard Nhl=en_US&gl=US Patreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: ffmprovisr https://amiaopensource.github.io/ffmprovisr/Jason:Pandas read_ods() read_excel()Topic: Differential EquationsWhy should programmers learn about DiffEqLaw of Large NumbersWhat are differential equations?When you know the rate of changeEigenVectors & EigenValuesWhat is Jacobian What is Jacobian? | The right way of thinking derivatives and integralsSpecial casesPartial Differential EquationsOrdinary Differential EquationsWhy solvers are importantNumerical Stability at larger step sizesExample: https://medium.com/@pukumarathe/eulers-method-and-runge-kutta-4th-order-method-in-python-b4a0068a8ebe Fun ExamplesPredator-Prey relationships in scipyhttps://scientific-python.readthedocs.io/en/latest/notebooks_rst/3_Ordinary_Differential_Equations/02_Examples/Lotka_Volterra_model.html Physics Engines for gameshttps://youtu.be/52n2qKgwW_Q PageRankhttps://arxiv.org/pdf/2001.08973.pdf  ★ Support this podcast on Patreon ★

programming throwdown episode 165 differential equations take it away jason hey everybody um this is going to be a cool show we're going to talk about something that patrick and i are definitely not experts in but i've had to do a lot of it recently and i think it's very important and so we're going to talk about diff eq um oh man you already shortened it i'm already out you know i don't know if uh did we have to take diffiffie Q in college? I did. Was it a requirement?

It was a requirement. I did very, very bad at it. You know, I honestly don't even remember taking it. So that goes to say, show something right there. I don't know if I took it or not, but we are going to dive deep into it. It turns out it is really important. And so we'll get to that. But before we do, I want to talk about spreadsheets. I actually fell back in love with Excel, in this case, Google Sheets. But we really take it for granted. I mean,

sometimes we even make fun of people who do tons of things in Excel. We say like, oh, we could have done so much of this in python or r or what have you but um you know one of the really powerful things in excel is the sort of like recursive nature of it and how you can kind of incrementally build on top of um some data and you can see all the intermediate values as long as you don't make the functions too complicated. So I had this thing where it's actually, it's kind of, it's pretty nerdy,

but there's this video game called X4. And in X4, it's one of these grand strategy games. So you mine ore ore use ore to like make solar arrays and solar arrays give you energy and you can use the energy in the ore to make spaceships and anyway it goes on forever and i thought someone posted a google sheet of all the different resources and some stats on them and uh for the resources that you know require you to build things like solar arrays you know what are the ingredients okay so i had this google sheet as you know material um the cost to acquire it if it's some raw material

or if it's not a raw material the time to build it and then all the requirements right and so um using some excel functions i was able to like recursively figure out the time it takes the total time it takes to get any ingredient so it's like the time it takes to build that thing, plus the time it takes to build all of its components, you know, times however many of those you need. So a solar array, it takes 60 seconds at your base time, plus, you know, it needs 10 of this other product. So whatever the total cumulative time is of the other product times 10 plus this plus this, right? And so that other product depends on other products and so it's this recursive thing but you know because i just specify and we talked about recursion in a couple of shows ago what was a pretty good tie-in um so you know because i just put this recursive step in and then i just control dragged you know

that equation onto every row and excel like instantly figured it all out and said oh for you to build you know medium-sized battle cruiser it actually takes like 18 000 seconds or something you know including all the other things you have to build and then you cried because you had already done it twice yeah i mean it's it's uh it was so you know what what what it made me think about is like that that's really cool you know like i just specified this simple recursive step and then excel you know handles like not calling the function too many times and and caching the values it kind of did all of it for me and i wonder if like a python library that does something like that would be really useful like some extension

to numpy or something where in certain values in the array you could put equations and and then python would just do what excel does like it would just figure it all out for you yeah i ran into something not as complex as what you were doing but we were talking about index investing a bit uh two episodes ago again oh man look at us tying it in good us yeah look at this the but i had this thing where you know some time goes by and then you want to put some more money in but you want to rebalance what's there? So the idea of rebalancing is you have some target. I want the most common, you know, one that gets toted about is 60% stock and 40% bonds, you know, go read up on that. I'm not actually advising you to do that, but let's say you want to do that. So you have some index that is your stock allocation and some that is your bond. And let's just say you have two for simplicity.

Then over time, one goes up and one goes down or both go up, and you want to get back to your, you know, 6040 target. But you also want to bring in some new money. So could you run this on a calculator for two? Yeah, it's really not that bad. But just dropping them in a in this case, I think Excel has extensions to do it, but I was doing it in the Google version, Google Sheets, and they have a, you know, go get the latest stock price. So as long as if I updated it each time I do it, then it knows how many shares I have. So it'll tell me the new price, I say how much money I want to put in, and then it basically calculates my trades for me. And if I want to, for instance, like disallow selling, like I don't want to sell something, I just want to buy new, then you know, you can do

that with the conditional logic. And sort of like you you're saying you iterate back on it and sort of you know step on it do not even iterations but just one column to the next where you're sort of rolling for your computations and it was just like oh because it turns out it wasn't doing it with two i was doing with like five uh five different indexes and so you know this is so much nicer to like set it up once and then be able to just rerun it if i had sat down in python i know there's an api for pulling down stock prices i know i could have put my stuff in a you know csv or a json and then read it in and done it but i i'm not clear that i wanted additional flexibility i'm you know i'm not trying to

solve this for every person or every case this isn isn't my job. I just wanted a quick solution. But I will agree with you that my hobby horses on this are people don't want to learn SQL. And we talked about that before on the show. But I think people don't like to learn spreadsheets. There's a group of people who just don't know how to run Excel or Sheets. And they don't know how to get graphs really easily. And they say, oh, it's OK. I do it in Python.

It's like, but I sort of guarantee if you only make this chart once or only for one thing you know doing it first in the spreadsheet and then if you go to do it again fine i'll give it to you that the second time you have to do it you can do it in python but that first time you almost guaranteed if you know what you're doing you're going to get it faster in the spreadsheet just dumping out some output you know new line separated dump from your program and then you can copy paste that into Excel and get your plot much, much, much quicker. Yep. Yeah, totally. And in your case, you're talking about balancing. So, you know, if you had a certain allocation, you know, a certain state, then you know, like how much of something to buy or sell to get you to equilibrium right so you have actually

a differential equation you have an equation that says here's how i get from where i am to where i want to be but then if you take a really large step there you might end up buying like way too many bonds and then you're unstable or something okay well before we get into that we will uh two episodes ago in our duo we were talking about the buzz and excitement of lk99 yeah and uh feels it feels like the what is the meme it's back and then it's over and it's back uh and uh if you if you followed it on x i'm just gonna go with twitter if you followed on twitter at all you saw these memes go by but um you know i it seems that a lot of people who sort of bided their time and really thought through it and did some analysis are pretty

convinced that at least at this point there's no evidence yet that that lk99 is is a superconductor so there's a nature article now out sort of saying this and some sort of well more well-reviewed stuff is out people are that the aftermath is somewhat interesting there's still some studying to be done of the material which you know happened but then there's it's it's very divisive i'm not a scientific scientist by trade or even a scientist period i guess um but there seems to be a big divide between was this a okay thing that it happened this way the original article was a pre-print right so it wasn't actually in a peer-reviewed journal it was just a pre-print

that everyone got excited about was this a waste of resources and you know does someone lose reputation or is this yeah this is just a more open, involved, broader scientific community and hobbyist sort of thing. I'll say from the sidelines, that's the most excitement I had had about material science in a preprint article ever. And I learned a lot. Clearly, there was a lot of speculation going on that we talked about a little last time. People saying things that just weren't going to happen or weren't true uh even if it had been everything it promised but yeah so what like you know i i don't i don't personally see what was harmed if people decided to try to pursue replicating this using resources that they otherwise shouldn't

have i feel like that's a that's a game theoretical decision that they made uh and it's it's sort of case by case so i i don't i don't sort of hold the broader community but again not my my domain so it's possible that i'm missing some part of the equation there yeah i mean i think that that uh it's a good question you i have a couple of things here one is i've seen some people say that okay it's not a super conductor but it's still important because it you know important because it's like a better conductor and there's certain things that we learned because of it. And so it seems like it is some kind of incremental step forward. The other part of it is, you know, there's definitely like a outrage economy. And so, you know, I think on things like Twitter,

you know, like a lot of people probably got a lot of followers for being, you know, really passionate about it. And so that's a really big, big problem, I think, that, you know, it's hard to overcome. Do you know if they know if they're still, like, is that material useful, even if it's not a superconductor?

Yeah, I think some people think it might be interesting, but once the hubbub has sort of died, I think a lot of people became a little more soft-spoken about things. Yeah, right. Like anything anything once it becomes more subtle right the you know it it's a little harder for people outside to maybe understand the implications i expect people will still continue some amount of studying down this

avenue um it seems like there's some novel properties but yeah i think the unlock of a actual room temperature and to be clear it's room temperature and room pressure like you know sort of normal atmospheric pressure um the the two thing missing either one of those sort of makes it just not the step function of a new discovery so even if it's still a superconductor at low temperature which isn't clear that like it could be a a massive improvement but not getting to where we are you know where the speculation was going sort of a letdown and maybe that's where people are pointing out is because it's a letdown of the public then the public loses faith in the in the process yeah that makes sense um my article is on normalizing flows have you ever heard of this i know this these words in individually in many contexts but i do not know what this is about

all right so i'll explain so um we have you know in in random math right we have probability density functions right and so what this means is you know if in random math, right, we have probability density functions, right? And so what this means is, you know, if you look at like a coin, a coin has heads or tails, right? And so it has a 50-50 chance if it's a fair coin of being heads or tails. If you look at a die, you know, a die has one through six printed on it. And when you roll it, you know, if it's a fair die, you have an equal chance of getting one through six, right? Well,

how do you do this with random numbers? And so the random numbers, you do it through probability density functions, which are like actually really non-intuitive. Maybe like a good example is like voltage coming into your house. So, you know, you have this vault meter and it measures the voltage, a multimeter, and it measures the voltage coming into your house, and it says 120. But it's not like exactly 120, or like exactly this number of electrons past the multimeter, it's some kind of rolling average of some, know estimate and so if you were to like dive into the multimeter it's probably

getting 119 121 you know and these are going to be floating point numbers like 119.x 120.y right and it's averaging them out and saying 120 um and so what you really have in your house, the voltage in your house is going to be some density function where the mean is 120 and then it has some tails. Like sometimes when you read it, you get numbers that are greater or less than 120. Right. Does that make sense? Yep. Okay. So it turns out, you know, if you have an expected value and there's just some error on either side, you know, normal distribution does a pretty good job.

And those are pretty easy to calculate. Right. What if you have a really weird thing? Like, for example, what if I have a data set that has American voltage data and European voltage data in the same data set so like you know half the numbers seem to be around 120 but then half the numbers seem to be around 230 and they're all just mixed together right so if you just fit a regular normal distribution to that you're gonna it's gonna tell you oh like almost all your voltages are around like 180 or something plus or minus some error which is not really representing what you

have what you actually have is what's called a bimodal distribution where you have one mode you have one cluster around 120 and you have another cluster around 230 um so you know what you want to do is you want to know the distribution of these kinds of data sets, right? These complicated data sets. And once you have the distribution, there's all sorts of interesting things you could do. Like, for example, you know, if you know the distribution of the voltage in your house, and let's say it's 120 plus or minus one volt right then if the voltage is like 130 or 150 you know like oh my house is really messed up like i like i'm gonna like my electronics are gonna start blowing up i need to do something about it right if you measure the voltage in

your house and it's like 120 plus or minus 20 volts and everything in your house is still working then you know like okay i have a pretty good tolerance for voltage in my house right um and so when you get to these really weird distributions you still want to know the same thing which is like have i seen something like this before and so to answer you know have I seen something like this before questions, you need to have a density function. You know, like how dense is this part of the system? How dense is this function? And so normalizing flows are a way to get like almost arbitrarily complex density functions so if you have really weird shapes that you know are are um you know that that you can't really categorize as with any normal density function you could always run

normalizing flows and it'll it'll learn that function for you um and so it's a bunch of really cool things you can do with something like this like i'll give you an example um you know there's these like breast cancer data sets right and and uh um they're pretty famous they came out of uh uc irvine and it's basically a whole bunch of data of like stats on on people and then i think there's maybe pictures mammogram pictures and then there's like whether there's a tumor or not. And, you know, you could train a classifier and say like, yes, breast cancer, no breast cancer. And there's a bunch of tutorials on how to do this on the internet. But, you know, you could use something like normalizing flows to say like, have I seen a candidate like this before? And if the answer is no,

then maybe you don't trust the screening as much. So if the screening says cancer, but you've never really seen anyone like this who had all these qualities, then maybe you ran an additional test. But if it says cancer and you've seen like tons of people just like this and they were all cancer,

then you're much more like certain, right? And yeah, and so there's a whole rabbit hole there but check out the link it has a really good explanation of how to do this and um this uh is a relatively recent thing i mean it used to be that you would um take that normal distribution the one i explained before, and you do what's called mixtures. And so mixtures are good if like the example I gave or something's either 120 or 230. So if you had something that was either like one of several values just with some error, then that's fine. But if something's just like arbitrarily distributed like it's just like some weird shape there's not really any way to recover that um until some of these recent methods

so i've been keeping an eye on it and it's they've been making tons of progress this isn't even the latest one it's just the one that i found the best explanation for that's relatively modern but i think that there's going to be a good future to these methods yeah this this is interesting i naively would have thought yeah i have come across this problem before i didn't know this was done this way and what i did is sort of you think about histogramming and then just taking the data you do have and sort of making i guess you would almost call it a discrete density function and just using the data you do have to estimate the percentages which of course

is difficult for things that once you've done the histogramming like things that come in between right but yeah there's there's some balance there but that's that's actually really interesting yeah i mean if you've ever built a heat map, I mean, that's basically a density function. And so this kind of takes it to the continuous space. Very cool. So my next one is probably going to require a bit of explanation, but it's how is Lama dot CPP possible? So Lama is and Jason, you're gonna have to help me if I get the background here wrong,

but Llama is one of the sort of chat GPT equivalents that came out of Facebook. And importantly, Facebook released the parameter weights as well as the architecture. And they just recently announced Llama 2. And there's sort of a debate around, is it open source? Is it not open

source uh they have some wording go read your go read the contract yourself the answer is for most people that aren't like a already established big company you can pretty much do whatever you want with it um and there has been a a person a group of people who had come out with this tool llama cpp which is basically uh implementing without uh i don't actually know i guess llama is in probably pi torch or something similar but without any of those frameworks from just sort of bare metal c++ implementing all of the required tensor operations and you know things that you need to do to take the weights and to take inputs and to get the expected outputs and the reason this is really interesting

is if you follow the space at all, the expectation of running Lama is that you do it on either some specific tensor hardware, like a TPU, or whatever, or more commonly on a very now expensive from a series of sort of circumstances, very expensive GPUs. And the biggest thing is you want to think about transferring the model weights to the GPUs memory. And then it allows all these operations to take take part very, very, very quickly. And this article sort of goes into this explanation that this is mostly about memory bandwidth, right? So the GPU has very expensive, very high bandwidth memory as well as discrete parallel processing units for doing all of the operations needed mama cpp doesn't benefit from you know sort of all of that gpu work what they are trying to do is using simd operations so it'd be like sse neon avx these kinds of things, which allow you to use your CPUs parallelization, but also using

the quantized model. So instead of floating point or double parameter weights, they do some retraining for a lot of these to get them down to four bits, or eight bits per parameter. And these have billions of parameters, I think most of them are like sort of like 12 billion parameters, right? So you're talking about a huge reduction in the amount of memory and then running them on just basically your CPU. Now, albeit a sort of accelerated thing, but the nice thing is you normally have a lot more memory

in your system RAM, just normal sort of like DDR4 RAM or whatever, you know, tied to your main CPU than you do on your GPU. So most people would not be surprised to hear 32 gigabytes or 64 gigabytes or even more of system RAM, but GPU RAM, you're probably, you know, 24 gigabytes would be really big, even 12 gigabytes is sort of large. And so a lot of people don't have a GPU is big enough to do this, but they do have CPUs. And so if you're

willing to lose some performance due to the quantization, although not as much as I was expecting, when you say I'm going to use four bits per parameter, I assumed all this thing's going to be just degenerate, but that's not true. It actually retains a lot of its functionality, which is fascinating.

Yeah, definitely. Yeah, then you can do it on your CPU. And the token speed, sort of the amount of tokenization that occurs is still you know kind of only like maybe one per second or that kind of thing um so not enough for sort of like that interactive chat feeling exactly but also not bad for not having to go out and buy you know a thousand dollar gpu uh and power supplies and everything to go with it yeah this is amazing super exciting um uh yeah i think it's uh it's really cool there's there's a

um there's a ton of of uh like uh posts about this and yeah the four-bit quantization pretty uh is pretty impressive i thought that you know with just what 16 numbers to represent each weight that you would have lost something really important but sure but yeah it uh it works really well um yeah i actually have with the four bit quantization i have a llama with seven billion parameters that runs real time on my phone oh yeah that's cool yeah there's an app called mlc chat this is for android i'm not sure if they have it for ios but you can run mlc you can install mlc chat and then you get a 7 billion parameter model right there on your phone it doesn't go to the internet or anything it's pretty remarkable and it um it will definitely hallucinate i asked it, what was the capital of Sri Lanka? And it made up a town. It might be another town, but it's not the capital. So it'll definitely hallucinate. You have to be careful. But I asked it how much Pittsburgh Pirates baseball tickets cost, and it nailed it. So, I mean, these are like pretty esoteric questions.

You basically have the power of a Google on mushrooms on your phone. And so it's pretty amazing. The thing that's interesting, I guess, is if you, someone was having this debate like, oh, if you're going to be stranded on a desert island is it better to would you take a usb stick with is a terrible question but usb stick for survival of like chat gpt and what it knows and doesn't know or a compressed wikipedia oh interesting probably probably chat gpt right what do you think i i don't i mean right now probably wikipedia but given like if we sort of assume things continue to improve i i think that

might start to change well actually the wikipedia one would you get a search engine too or do you have to like i think so i think oh okay that's kind of a game changer okay yes i'd probably go wikipedia but i think the interesting thing is chat gpt would potentially know how to surface in a more interactive queryable manner or even distill something so you know we were talking about you're talking about normalizing flows right and some some stuff if i had questions about this going to wikipedia might quickly go over my head and it's going to take me a very very very long time to get to where i need to be to understand versus you know chat gpt could synthesize that down for me and potentially explain it the problem like you mentioned is right now it's sort of not self-aware of when it's hallucinating and so you could potentially end up getting an answer that it's very confident but very wrong yeah i also don't know if you

described something right like if i described a coconut to chat gpt maybe it would be able to tell me it's a coconut it would be very difficult to query wikipedia for finding out you know is this what is this thing i found on my desert island and can i eat it yeah i mean i was thinking about um building a device like this my idea was basically a flashlight that had um an open source large language model and uh speech to text and text to speech so it's like you could hold the record button and say you know is this tree with four leaves poison ivy and then it would say something back to you that that is really cool although you could also use it to start a fire with the heat it generates so you know like perform two two roles oh man um all right so actually uh my news is also related to LLMs. It's this cool website called chat.lmsys.org. And you can go there, you can ask questions to a variety of different open source, large language models, and you can even ask questions to multiple at a time and see what the different answers are like

on the spectrum there. So I typed in to the 13 billion Lama two model, what are normalizing flows? And it nailed it. It said normalizing flows are a type of generative model. They use a series of invertible transformations to model the distribution of the data. And it,

it went into the different types of flows um it doesn't talk about radial flows which are the most popular so it missed out it listed four flows but in my opinion missed out on the most important flow um so you know in general i think that you know that's kind of in line with what you'd expect where like some of the details are kind of hazy, but it gets the overall idea, right? Um, yeah, I mean, this thing nailed it. It explains why it needs to be invertible. Um, really impressive.

Um, so yeah, check this out. If, uh, um, if you, if you haven't really been able to access open source models it seemed like it's kind of too far from reach technically um this is a super easy website you literally just click and it's basically the chat gpt interface but for open source models and so you know you can play around with this it's using someone else's compute so it's totally free maybe at some point you'll get rate limited. I haven't tried that. I haven't gotten there yet.

But you can just play around with this. And if it solves something for you, then that could justify taking the time and energy to get one of these running on your own computer. Also, this LMSYS is all open source. So you can actually clone this repository and make your own version of this where on your computer,

you can ask yourself questions or your hardware questions. So you could go from this chat GBT interface to something running on your own hardware, which is pretty exciting um i've i've talked to folks who are generating real value with uh chat gbt um chat gbt is so cheap it's really hard to justify you know the the open source models um you know A lot of the folks I talked to are just using ChatGPT. My interactive fiction game, I have a bunch of folks playing it, which is really cool. And even

with all the folks playing it, last month, my bill was 57 cents. So it's hard to compete with that. But yeah, I mean, I think for sensitive things or as a curiosity or or for whatever reason if you want to do the open source ones this is a great opportunity to see how they compare and how they how they've been maturing time for book of the show what is your book of the show patrick my book of the show is uh as as uh is adequate for my level of thinking math with bad drawings um by ben so i have not i have not dug all the way into this book i i have this book and uh i i am familiar with some of the writing here and you know i've read a little bit of it but for me having uh you know i'm not a

mathematician by background or by training in fact i did very poorly at it uh spoiler alert for the upcoming topic but i will say that like i do end up having to do a fair bit of math as i think many people in computer science end up at one point or another or you know i guess maybe depending on your your sort of role but i do do a fair bit of math but for me it has to make sense to me and it's useful to find people who at minimum find alternative ways of explaining but i will say find non-academic ways of explaining intuition behind things it is sometimes hard to to sort of explain arbitrary topics via intuition from start to finish but there's a number of people I'm

very happy to, to have found. Um, uh, and we took about one later, but there's like three blue, one Brown is like a YouTube channel we've talked about before, or he does these math explanations, even watching, um, you know, just math videos on YouTube in general that aren't, aren't classes are good, but Ben Orland has written this book and sort of covers a variety of topics. There's several follow-ons in the same thing. One about playing games, one about, uh, sort of, uh, calculus and just sort of some comic just to kind of like keep you engaged, but also just sort of like cheesy illustrations, um, in order to, to sort of help you help you think about it.

And it just vibrates sometimes with how I think about the world. And so, uh, I recommend this book if you're at all curious, if you're a math tourist, I would say, who's just interested in kind of like a light treatment, a popular treatment, I guess, of a lot of these topics, definitely check it out. Very cool. Actually, you sold me on it. I literally just bought it while you were talking. Oh, no. It's going to be a book of the show next time for Jason. It's going to be my book of the show. We were talking about this before the show, how sometimes a lot of the book I'm reading

right now is kind of invalidated because it was Patrick's book of the show like seven years ago or something. But it doesn't mean you shouldn't read it. Definitely go back and and read all of our books of the show a lot of them are real winners um all right my book of the show is uh beyond reading brandon sanderson which we talked about a couple episodes ago um i was playing this text adventure called overboard it's from the people who made 80 Days, which was my book of the show in 2017. This one is interesting. The challenge with interactive fiction is replayability,

right? I mean, clearly, like a book has, you know, very little replayability compared to video games, a regular book. And so interactive fiction, you have to kind of figure out, if you have to create all this content for all these things a person could do, but then they only do one path and then leave it, then a lot of that content might not be explored by most people. So they want you to be able to play the game multiple times. And this one takes an interesting approach where they have a system of achievements. So you play the same game every time.

There are multiple endings, but a lot of the same thing happens. It's deterministic. So if you do the same thing at the same point in time you'll end up with exactly the same ending um but as a way to keep you finding new content they have this system of achievements and the achievements are based on what you have and haven't seen so if you play the game and just coincidentally you get the best possible ending then you'll get an achievement for like kill as many people as possible you know be like the most evil person

possible because you haven't tried that yet um i thought that was really really clever um i feel like um it wasn't executed perfectly because um it's not clear how to get some of these achievements so you see the achievement and maybe you play you don't get the achievement you have to keep trying and trying and while you're trying you're just trying to do random things it didn't do the best job of keeping track of what you've ever done in the past. So, you know, it has some work to do there and execution, but I love the idea. I think this idea of achievements kind of takes interactive fiction and makes it more like a roguelike where, you know,

you start the game with like advantages the more time you play it. So I think there's something really powerful there on the game design front. And I had a lot of fun it's a good story and so uh um let me just check really quickly what it costs um i can't see what it costs because i already own it um but yeah it's probably like five bucks totally worth it um uh highly recommended it 649 definitely it's worth you you'll easily get a few hours of entertainment minimum out of it so and if you don't want to uh buy that game uh you can uh actually buying that game doesn't support us because it's not an amazon book but if you don't want to buy the math with bad drawings book,

like I just did, you can also support us on Patreon. So if you go to patreon.com slash programming throwdown, big shout out to our supporters. We have had an influx of supporters, which is really cool. I wonder if that's companies forcing people back in the office something to do with that. I don't know. If you are forced back into the office and that caused you to support us on Patreon, we lament the fact that you're forced to go back in, but we really do appreciate the support. All of that money goes in an account, which we use for the show. We don't really pocket we don't actually literally don't pocket any of it we use it to try to bring more people uh into the show and spread the word

so we really appreciate every every dollar of support that we get there and with that we'll go oh go ahead patrick oh no just saying thanks you go ahead oh yeah thanks thanks everyone out there we'll go to tool of the show um why don't you go first okay mine is a website uh ff improviser uh good luck trying to google that you could just but this is it that's not even the full website this is a github link to a sort of interactive website with i guess what you call, almost FFmpeg one-liners. So I've had FFmpeg as my tool of the show before Jason did. One of us did. Super powerful tool.

Horrible command line interface. I'm sure it has to be. It probably has to be. I'm not saying I could do better. To be clear, please don't at me. Well, good luck. You can at me. I don't see them.

Yeah, you don't have an at. I didn't know where you would at me at. But anyways, the benefits. Okay, sorry. You can send me a letter if you could no no no don't don't yeah um all right so ffm provisor has a bunch of things you may want to do with ffm peg and gives you sort of one-liners that you can use to combine together and so definitely makes a very powerful tool but hard to use a little easier to use you know helpful um so definitely check it out i also am bumping into this i did see recommendations for a group of similar tools include including some like visual if you ever played with like lab view

like you drop little modules in on like a data flow graph and it'll sort of generate a FFM peg commands for you. So shout out to a bunch of people like out there trying to, you know, make, make ways of a FFM peg easier to use because truly a powerful tool. I mean, it's used all over the place. This is really awesome.

Video and video codecs are insane. I, I, I still like don't even profess to understand what any of them do or are or the difference between container versus you know i'm not even gonna say it anyways so definitely check it out ffmpeg improviser ff improviser uh very useful tool for for helping you because if you're like me uh i yes never remember i have like little read me sprinkle all over the place for places i normally run ffmpeg to try to remind

me of of the command i used last time nice very cool um my tool of the show is um pandas read ODS and read Excel functions. So I know that's kind of really specific. We have talked about pandas, but it's been probably, I think it was 2016 when we talked about pandas. Pandas is basically a library for getting data frames in Python. If you've ever used R, you know what a data frame is

if you haven't it's basically very similar to an excel table where you have columns each column has a name and a set of values and then you can do arithmetic you can add columns together and stuff Um, and pandas actually can read ODS, which is the, you know, uh, Excel file format for the open, open Excel file format. So you can save a Google sheet to ODS, obviously LibreOffice and OpenOffice read and write ODS. Uh, Excel probably will save ODS as well. Um, pandas can also read Excel files like the XLS and XLSX and all those files. And what Pandas will do is it will actually evaluate all of the formulas,

resolve them to their number. What it won't do is it won't on the fly evaluate the formulas or anything. That's something we talked about at the beginning of the show but but if you have an excel spreadsheet with a bunch of complicated equations and you want to bring that data that output into python you can just read it in one line using pandas which is pretty remarkable so um um so yeah i think there's a uh i think there's a good synergy there um and uh yeah i was really impressed when i found that function that is very cool i yeah the fact that you said it even does like evaluation not just like reading it as columnar data is is pretty cool yeah i'm not sure like oh go ahead i was gonna say i want to make a joke about like VB script and whether it evaluates all the random macros, but.

Oh God, no. You know, I actually don't know if the, I wonder if the file format saves the function and the value, you know what I mean? In the file or if pandas is doing all the calculation. I really don't know i will say i've never written a complex enough equation in my spreadsheets that like i would have been able to tell the difference of whether it was like cached or computed you know sort of like them i i would like when you open the file does it take i guess if you could construct processing complicated enough that it takes literal minutes to sort of finish computing

your spreadsheet in which case it would make sense that they write the value out as well right as a cache oh yeah but so it'd be easy enough to test i've just never written such spreadsheets myself so you know my guess is it probably has the cache value because otherwise the pandas people would have to implement whatever library they use, would have to implement like every Excel function and test it to make sure you got exactly the same answer. It's probably one of those things, you know, it's almost non-existent, the number of Excel files with a lot of the very random esoteric functions. So it's like, do they support 99% of the files? Five, nine, six nine six nines like most excel sheets probably don't even have a formula in them yeah it's a good point it's a really good point um but either way

give it a shot um you know take it uh i mean panda is extremely well supported but still i would i would always have like a little degree of caution with something like that because you are going from one language to another yeah there's probably like a decent way to to like force it into a good state even if you have a tail tail spreadsheet right copy the values out or something and you lose the formulas but pandas would still be able to read it yep yeah. Time for differential equations. So we talked about recursive functions scaring new programmers, and now we have something that scares us. I'm actually pretty nervous, dude. I'm shaking right now.

So let's start, as we always do, with the motivation. Why people should learn about DiffEQ. So usually when you hear about differential equations, you think about physics or maybe economics. Actually, we'll put economics aside for a minute. You think of like physics or these kind of like signal processing, these kind of things that are really kind of low level, things that interact with the real world through sensors and all of that. But there is a reason why a lot of people, even if you're building, you know, a website or something like that, or you're driving engagement to a website or doing something, you'll need differential equations.

And that is because of this thing called the law of large numbers. Have you heard of the law of large numbers? Not in this context. So normally I think of it in statistics. So I'm excited to see if it's the same or not. It's exactly the same. So the law of large numbers says, if you have a lot of things and you need to reduce those things, like let's look at dice. If you say, I want to throw 17 dice and I want to get the average, right? So you can just keep track of all these numbers and take the average and you're going to get something, right? What if you get to millions of dice or billions of dice you know what can you do what what the law of large numbers says is that if i average and you know a sequence of numbers from a distribution that i'm that that average is going to be centered around some point right and so with dice for example, you know, if you average a lot of die rolls, you'll get something around 3.5.

Right. If you do it enough times, you're going to get around 3.5. Even if like the first 20 times you rolled, you just coincidentally got a one. You know, eventually with enough samples, the odds of that kind of anomaly go infinitely close to zero. And so you will end up with a number around 3.5. And so, you know, if you want to do things like how much time are people spending on my website or how many people a month are visiting my website or maybe a month's bad example, how many people a minute are visiting my website or maybe a month's bad example how many people a minute are visiting my website so anything where you're going to end up with many many many many many samples um you're going to end up probably doing an average but you know taking some kind of estimate of these samples so you're not working with billions of numbers and as soon as you do that, now you have a continuous number. And so if you, if you were thinking about like traffic coming into Google, right, they're not, they're not looking at it in terms of like, okay, I have this exactly this many people and then exactly this

many people. And I'm going to draw some insights. Like, no, they're taking averages and they're looking at sort of trends over those averages and they're constructing some type of differential equation that's saying, look, people are, you know, this metric is going up, this metric is going down. Here's the covariance, which means as one metric changes changes how it varies the other metrics and so based on the velocities of these metrics and the covariance of all of these metrics you can construct a differential equation that shows you like how you think these metrics are going to evolve in the future. Yeah. So looking at the, the, the, oh, I'm going to start saying words we're not supposed to, but anyways, looking at the, the rate at which the people are coming out and then sort of understanding how those change over time. Yep. Yep. Yeah,

exactly. And so, you know, there's many situations where you have what you want to achieve in the next time step, and that is easy to compute. I think the balancing stocks was a great example where it's like, if I have 60% stocks and 40% bonds and I want it to be 50-50, well, then I know what I need to do. I need to sell stocks and buy bonds until I hit 50-50. And so you have your difference from your goal to your current state that's relatively easy. What's hard is sort of keeping that stable so yeah and this is why they pop up in uh if you ever read stuff about controls and control systems so yeah trying to figure out how to turn the steering wheel on your car to get to where you want to go quickly but not overs. And how do you sort of think about computing these

things? Yep. Yeah, exactly. So like, let's say you had, um, so let's say you, you, you know, you, you like in this case, it's clear. It's like, okay, there's a percent. And so if I get 1% of one, it's kind of one less percent to the other. Well, let's say you didn't really know that the stocks and the bonds had to add up to 100%, right? So you might say, well, the stock number is 40%. I need it to be 50%. So I'll buy 10% more stocks, but then you don't have, but then, you know, Oh, you won't, you won't have the exact 10%. So I'll just buy more stocks, but then you buy more stocks and then you overshoot or, or even a better example is, um, you say to yourself, well, my house is 80 degrees. I need it to be 79 degrees. And so, you know, I'm just going to turn the AC on, you know, full blast.

But you do that and then it causes you to overshoot and you have to kind of keep going back and forth. So it turns out that for a lot of these, what you really want is something that is you know pretty stable and oh another part of this is it gets really complicated when things affect other things so for example let's say you have you know air conditioning and humidity but when you run the air conditioner it you know causes your house to become less humid. And so you might have a humidifier, but then the humidifier drops the temperature. And so these things are affecting each other because the differential equation like not long enough to support both of the cars and everything starts like jittering and then spiraling out of control. That's a case where, you know, the game, because, you know, it wants to run in real time, it's going to take, you know, kind of larger steps and it's going to spiral out of control.

So, yeah, we should actually talk a little bit about step sizes. So, you know, in Patrick's example of the stocks, right, you might say like buy 10% more stocks. So let's say you instantly do that. And, you know, Patrick is a billionaire, right? So that's not a small transaction. So that's not a small transaction. So 10% of all of that, let's say that influences the stock market, or even just it's happening at a really volatile time where stocks and bonds are going up and down. It's crazy. You buy that 10%, but you buying that kind of changes the market dynamics. Maybe it makes the price go up. And then also other crazy things are happening. And so you find that, oh, after I did that buy, I'm actually even further from the goal. It's like being bad at mini golf. You're bad at mini golf and you putt and it goes past the hole and then you're even further

from the hole and you putt again and it's just getting worse. So one way to do that is to say, well, here's what I'm going to do. I'm going to buy one share of a stock or a bond. Let's say buy one share. It's going to get me a little closer to 50-50. And then I'll measure again. It's like, okay, now I'm at 40.1% stock bonds. So I'll buy one more bond. I'm at 40.2%. I'll buy one more bond. And so you could do that and you'll eventually get equilibrium and it'll be stable, but it will take forever. Right? right and so that's the paradox is if you go really slowly then then uh it takes a ton of either compute or time or what have you um if you go quickly then you can cause like second order multi-order effects um and there's an example i have that folks should check out um in the fun example section at the bottom it's predator prey relationships so the idea there is um basically you know every fox

eats one deer um and deer multiply at a certain rate and uh the question is you know how do you end up with something stable so if you have 20 foxes and 20 deer and you take one full step, well, what happens? All 20 foxes eat all 20 deer and then all the foxes die because all the deer are gone. Right. So you can't just take one full step or it just destroys itself. Right. So you could take baby steps and say, OK again i'm going to do this i'm going to assume that because these law of large numbers of this these 20 foxes are actually you know 20

million foxes and so i can kind of divide them into smaller and smaller numbers and so you know the the 20 foxes start eating more deer but they don't all eat a whole deer there's just tiny micro changes and the deer population starts tanking and then that means the fox uh population starts dropping because they start starving and that allows the deer population to recover and then you get this sort of like harmonic situation and um you know basically you can see from this this site like based on the step size either you know all foxes eat all deer and then they all die or you get this like oscillation thing or uh with a small enough step size it kind of converges to some equilibrium where enough foxes are eating just enough deer to match the growth rate of the deer and it's all

kind of exact. And if you had started with that end state, it would be an equilibrium and you could set the step size to anything and it would just stick there. But because you started from a state that wasn't an equilibrium, the step size size really starts to matter does that make sense yeah totally and i think this is part of the the sort of modeling setup and the iterative nature of a lot of these things and and sort of running them forward and you already kind of mentioning but simulations where you you hear about this a lot and thinking about the the consequences of what you're doing and and how to simulate these things um i think with like what you're saying i tend to think of a more like discrete way right like oh you could have like the foxes be in cells and the deers i think there were foxes and deers

be in other cells um and then like having them move around and that is one way of doing it but i guess you call it more like an agent simulation but that's is one way of doing it but i guess you call it more like an agent simulation but that's the only way to do it you don't actually need to model any individuals where you're kind of saying by the law of large numbers you don't need to actually model any individual deer or any individual predator like it you could just model it at scale and talk about the full thing where doing an agent-based simulation actually would run into a lot of problems so they're sort of treating the problem with uh different approaches yep yeah exactly and so you know a lot of biologists i mean definitely biologists like the 1900 or 1800s and stuff you know they couldn't do the simulation and so they had to rely on these really coarse things where

they'd have people go out and like you know count the number of foxes in like a whole county and then, uh, do that a few times until they can get some kind of statistical average. You know, they could use the law of large numbers, even if they don't have a lot of numbers. Right. And so just tolerate some kind of error there and then, and then move into differential equations. Um, so yeah so i think um okay yeah so we'll dive into um you you'll hear the word ordinary differential equations and partial differential equations so so if you look at like the fox example you know the fox the number of foxes affected by the number of deers and and vice versa and all of that um but you don't really care about modeling it in those dimensions the only dimension you really care about is time

so it's like you know what is going to happen in the future? So, you know, if F is number of foxes, D is number of deer and time is T, you know, T is the only one you're exploring. Now F and D are changing as you, you know, move through time, but you're not moving through the number of foxes. You know, you, you set an initial value and then you move through time. And so what that means is you're only differentiating through time. So that's where partial differential equation comes from. The equation has many different variables, but you're only interested in a part of them for the purpose of differentiating. And so if you're only differentiating through one variable, and there's maybe some other conditions too, then you have an ordinary

differential equation. And if you have an ODE, then a whole bunch of techniques open up to you. Another thing about, I think, ordinary differential equation is there's no discontinuities so this is this is again a really uh point that folks can get stuck on right they might say well you know i might have some if statement that says well like if i'm in a certain case then the foxes eat the deer otherwise the foxes don't eat the deer. And so there's this break, right? There's some variable that's like binary, right? And if you do that, then all of these techniques aren't going to work. And so that might sound really limiting.

But again, when you start looking at really large numbers, you should kind of expect everything to have some inertia right you shouldn't expect there to be like these really sharp discontinuities just like you know you wouldn't expect like everyone to leave google tomorrow like it's just not not really reasonable like even if google like somehow uh uh the whole page is full of ads tomorrow or something made some kind of really bad product decision there'd still be like this decay right so um so you know at this scale you know a lot of those constraints are pretty manageable and um um and so once you have an ordinary differential equation then there's a whole bunch of interesting things that pop up. Um, you know, in the way that we were talking about solving, you know, Patrick's balancing stock balancing issue, we were using

Euler's method. We were basically saying, you know, we know how to get exactly to the goal, but we know that if we took that big step that, um, you know, it would cause some disruption and it would move the goalpost and we would be in trouble. So we'll take baby steps. And at each step, we'll reevaluate the distance to the goal. Another good example of this is missile tracking. So if there's a target and you have a missile, you're playing some kind of video game or something. The missile might say like, oh, my target's here.

I need to draw like this arrow. Like I need to go in this direction. But then, you know, the target's moving. So as the missile is going to the target, the target's also moving. And so, you know, the missile would end up kind of curving towards it if, if it didn't have any predictive capability. Right. Um, and so that's, that's another example of a differential equation.

Um, oh yeah. So, so Euler is a first order method. Um, there's also like second, third, fourth order methods. And what these do is they look at a series of predictions and they use a series of these to get an even more accurate step. Um, so for example, um, this is using the mini golf example let's say you overshot the hole and uh um you kind of learn to yourself like okay you know i need to hit it a little less hard next time right so even though like maybe you overshot the hole and you're twice as far as you were last time you kind of learned okay i need to hit it twice as hard but also a little bit less because i clearly hit it too hard this time let's say you hit it and you hit it you know 1.5 times as hard and you still overshot the hole so you kind of learned like okay i need to hit it even less than

that and so with these other multi-order methods you can take bigger steps towards the goal and still have less of this overshooting problem because it's taking more information into account we were been talking about i guess like controls and like controlling things or taking action but i think we also see them pop up in sort of running physics engines and games and to me there it's a little different you're attempting to model the setup of the differential equations like the force of gravity on a ball falling down in your video game and there you're trying to say i'm actually wanting to compute position by given an arbitrary time so i'm going to give you some time in the future which is you know my game last displayed here and i've moved x

time step forward and setting up your equation so that instead of defining that step in advance you can compute the position at an arbitrary time into the future. And so there, I think to your point, you're sort of changing the situation a bit rather than what action am I going to take? You're sort of giving the system, hey, here's a new time. Can you tell me what all the positions are now? Yeah, that makes sense. Yeah, actually, it's a really good example. Like imagine if you have a ball that's moving towards a wall, you might say to yourself, like, this is a pretty simple situation. And I could take a pretty big step. In fact, if there's nothing around, maybe I take such a big step that the ball hits the wall. And I don't have to think about anything in between. So, so, you know, just maybe the ball is so far from any other object that you just say, look, for the next three seconds, this ball is going to arc and it's going to bounce off the wall and I can come back three seconds later. I don't have to do any math at all. Um, conversely,

like maybe the ball is chained to another ball and both of those balls are now like, you know, moving through the air air but in weird ways where they're kind of pulling on each other right well that is a really unstable and uh and so you might need a really low time step and all these physics engines do kind of dynamic time stepping and stuff like that but you know using a multi-order method is even better um if you can afford it so the equations are more complicated i thought you were gonna you said two things i thought you're gonna do three and we're gonna talk about the three body problem um but no you avoided that but i mean just to reference that i mean i think one of the things you'll see pop up when you're

you're talking about differential equations is if you think about, as Jason was mentioning, things interact with each other. So it's not just, you know, one object moving through the world without consequences. If you think about in space, if you think about two planets or two massive objects or low mass, anyways, it doesn't matter. Think about two things in space near each other. They are pulling on each other gravitationally. There's a force right so this is like you are trying to understand why does the moon orbit the earth but

the earth moon together also orbit the sun um and so if you sort of think about the interactions of these as they play out and fast forwarding through and understanding where they're going to be at some time point it it depending on the level of precision you care about yeah the moon is way smaller than the earth but it does actually cause the earth's orbit to wobble around as it spins around it's pulling on earth just as earth is pulling on it which is um you know you'll see things like oh we are predicting that there's a planet that we can't see because there's these slight disturbances in another orbit of a planet, you know, as something is passing by and they pull on each other. And, you know, that is sort of a continuous thing. And it doesn't matter how small or big it is.

Now, at some point, you probably just don't care about it anymore, right? You know, a baseball space debris floating around the Earth has an influence on its orbit, but it's such a degree you're probably going to ignore but as you sort of try to account for more and more of these at you know increasing levels of position a precision uh you know imagine trying to launch a space vehicle to meet up with mars and make a precision landing right you need to be able to solve these differential equations in a very precise manner as jason's pointing out like allowing to layer on more and more uh you know ways of increasing the accuracy and precision because you really care you really want to know the exact moment that the two things are going to

meet up because it has a range of consequences including how much fuel you put on your rocket and how much fuel you put on your rocket determines how much velocity it's going to have and it's a it's a cascade of things. And you need a setup that allows you to iteratively converge on a solution. Yeah, totally. Totally, totally. Yeah, I mean, there's so many things where when you start modeling at a high level, you run headfirst in differential equations.

Like imagine you're making some some kind of strategy game right so it might be that if you want to know if your game is fair if it's balanced um it might be hard to do that in simulation you know because you have all the different units and they're moving around and and it could be really difficult to think about all the all of that and compute all of that what you could do instead is you know come up with some mathematical models that say okay you know if i have uh so maybe i have a measure of army strength which is just a number that i've learned and i've learned that you know for this configuration of units i have a strength of this for this configuration of units I have a strength of that and I learned some function that says given two army strengths which one wins and maybe what's left of their army after they win

uh so given given two army strengths that fight you know what are the army strengths at the end one of them will be zero and the other one will be something um you know now you can construct sort of differential equations that talk about um you know should i you know if i like like what happens to my army strength as a function of the current base yeah am i like how is the army strength growing and subtracting and all of that? And so, you know, once you start abstracting out to kind of a higher level going to be full of partial differential equations. And I bet you even beyond what's in the game, you know, they were tuned and that game was tuned using differential equations solvers. You know, another like more kind of industrial example or commercial example is PageRank. So PageR rank was the algorithm that google used initially to rank all the content on the internet and the way it works is they manually uh give some amount of of

rank uh think of rank as like a resource right they give some amount of rank to a bunch of websites that they uh code up by hand and then they say when those websites link to a website um some of that rank is diffused onto that website so um so if, uh, you know, um, uh, like msn.com links to some, some, uh, kataku.com and some energy passes from one to the other. And so they used, initially they used Euler's method. So they said, well, we're not going to give just all of MSN's energy to like the first link we see. That would be crazy.

So we're going to look at like roughly how many pages does how many domains is this domain link to and will diffuse some portion of their energy equally to all of those pages. And they're linking to other pages and those pages are linking to other pages and so you have this multi-order effect and so um you have a you have a partial differential equation so as time progresses you know this this rank diffuses through all of these websites until eventually it stabilizes and using um you know differential equations and you know the theory behind that you can compute the page rank you know more or less efficiently while still getting a pretty stable answer and even you know when you're trying to figure out how much rank should i diffuse among the different pages and what's a sort And what's the most fair way to do that? Differential equations will allow you to do that very quickly. So for example, if you were to

just use Euler's method and compute page rank on the whole internet, maybe that would take days, maybe months. But if you were to use faster methods, you could get the same answer in like two minutes. And so getting it that quickly allows you to do a lot more interesting analysis we haven't yet told people how to actually solve differential equations jason oh man uh yeah that's a really good point i'm teasing i think this has been this has been a great sampler like uh an overlook of the various aspects. And I feel like it's one of those things where, you know, hearing it over audio, I guess maybe some people are out there furiously scribbling notes. But I feel like that's the right way to cover it to give sort of the list of things, the sort of what's the next hop in the journey to learn more about these things, to understand if you've encountered these things and didn't know the right words for it.

We were talking about that, right? About Wikipedia versus ChatGPT. How do you sort of find the next thing? So hopefully this has been useful to people. I mean, without going into a very lengthy discussion about how to actually run Euler's method or do some of these things I feel like this has been a great introduction to the topic yeah I think this is one of those categories where you should use software off the internet so sci-fi yeah like

sci-fi for python is amazing they have like seven different solvers they're all great um um but i would say the first thing you need to do is is come up with a function and you can pick whatever language you want but come up with a function that says given my variables so like using patrick's example you know given my percentage of stocks and bonds you know what is the step like what is the difference i need to make to solve that so in patrick's case you know if he has 40 one 60 the other then the difference is to get it to 50 50 and i guess you know you need the denominator for that but but you know let's say you pass that in it's like oh here's what i could do that would like instantly solve my problem based on what i know right now come up with that function make sure it doesn't have any side effects doesn't

need to like read from a file or anything it's some pretty small function it doesn't use globals or anything like that um and then you can plug it into a solver and you can actually explore different solvers and see what happens. Cool. Very cool. Yeah, definitely check it out. It's a good, it's an important thing. You will often be in a situation

where you know what needs to be done, but then you do it. And as you do it, you change the nature of the system. And it seems like you never really get to the goal. That's what differential equations are really good at fixing. So yeah, check it out. Check out the links. We have a ton of really good content in the show notes. You can go to programmingthrowdown.com or you can look at the show notes tab of your podcast app if you have that available you can also go on our patreon and we have all the show notes there and uh

and a super fast rss connection over there thank you everyone yeah all right thanks everybody my hurts. How do you feel? Is your brain on fire right now? I guess it's time for me to compute the rate of change of hormones in my brain from neural connections. I need to compute how hard my head hits the pillow. All right, everybody. Hopefully we didn't melt brains. It's a super interesting topic, and we will catch you all next time see you later music by eric barn dollar programming throwdown is distributed under a Creative Commons Attribution Share-A-Like 2.0 license.

You are free to share, copy, distribute, transmit the work, to remix, adapt the work, but you must provide an attribution to Patrick and I and share alike in kind.