The Changelog: Software Development, Open Source - GPT has entered the chat (Interview)

Episode Date: December 16, 2022

To wrap up the year we're talking about what's breaking the internet, again. Yes, we're talking about ChatGPT and we're joined by our good friend Shawn "swyx" Wang. Between his writings on L-Space Diaries and his AI notes repo on GitHub, we had a lot to cover around the world of AI and what might be coming in 2023. Also, we have one more show coming out before the end of the year — our 5th annual "State of the log" episode where Adam and Jerod look back at the year and talk through their favorite episodes of the year and feature voices from the community. So, stay tuned for that next week.

this week on the changelog well we thought a good way to wrap up the year will be to talk about what's breaking the internet again yes we're talking about chat gpt and we're joined by our good friend sean swix wang between his writings on l space diaries and his ai Notes repo on GitHub. We had a ton to cover around the world of AI and what might be coming in 2023. We do have one more show coming out next week before we wrap up the year. And that is our fifth annual State of the Log episode where Jared and I look back at the year and talk through all the great conversations we were to have. So stay tuned for that next week. A massive thank you to our friends at fastly and

fly fastly keeps our pods faster download globally because hey fastly is fast all over the world learn more at fastly.com and speaking of all over the world our out at fly.io. This episode is brought to you by fly.io. Fly's vision is that all apps and databases should run close to users. That's why they've made it their mission to provide globally distributed Postgres, MySQL, and SQLite with more databases like MongoDB and CockroachDB to come. They have generous free tiers everyone can use. Learn more at fly.io slash changelog and check out their speedrun and their excellent docs. Again, fly.io slash changelog. all right well we have have Sean Wang here again.

Swix, welcome back to the show. Thanks for having me back on. I have lost count of how many times, but I need to track my annual appearance on The Change Log. Is that twice this year on this show and then once on JS Party at least, right? Something like that. Yeah, yeah. I don't know.

It's a dream come true because, you know, I changed careers into tech listening to ChangeLog. So every time I'm asked on, I'm always we had Simon Wilson on the show to talk about stable diffusion, breaking the Internet. You've been tracking this stuff really closely. You even have a substack and you've got obsidian notes out there in the wild. And of course, you're learning in public. So whenever Swix is learning something, we're all kind of learning along with you, which is why we brought you back on. I actually included your stable diffusion 2.0 summary stuff in our ChangeLog news episode a couple of weeks back. And a really interesting part of that post that you have that I didn't talk about much, but I touched on, I want you to expand upon it here, is this idea of prompt

engineering, not as a cool thing, but really as a product smell. And when I first saw it, I was like, no, man, it's cool. And then I read your, just your explainer. And I'm like, no, he's right. This is kind of a smell. Dang it. He's right again. Yeah. We just learned about prompt engineering back in September with Simon and talking about casting spells and all this. And now it's like, well, you think it's overhyped. I'll stop prompting you and I'll just let you engineer an answer.

Well, so I don't know if you know, but the Substack itself got its start because I would listen to the Simon episode and I was like, no, no, no. Spellcasting is like not the way to view this thing. It's not something to glorify. And that's why I wrote multiverse, not metaverse, because the argument was that prompting is,

you know, you can view prompting as a window into a different universe with a different seed. And every seed, you know, is a different universe. And funny enough, there's a finite number of seeds because basically stable diffusion has a 512 by 512 space that determines the total number of seeds. So yeah, product engineering is product-sponsored. I have to say it's not my opinion. I'm just reporting what the AI thought leaders are already saying, and I just happen to agree with it, which is that it's very, very

brittle. The most interesting finding in the academic arena about product engineering is that default GPT-3, they ran it against some benchmarks and they came out with like a score of 17 out of a hundred, right? So that's pretty low benchmark of like just some logical deductive reasoning type intelligence tasks. But then you add the prompt, let's think step-by-step to it. And that increases the score from 17 to 83, which is extremely like, that sounds great. Like I said, it's a magic spell. I can just kind of throw onto any, any problem to make it think better. But if you think about it a little bit more and like, you know, would you actually use this in, in real work environment?

If you said the wrong thing and magically it's like suddenly deteriorates in quality, like that's not good. And that's not something that you want to, you want to have in any stable product. You want robustness. You want, uh, you want natural language understanding to understand what you want, not to react to random artifacts in keywords that you give. Since then, we actually now know why let's think that step-by-step is a magic keyword, by the way. And this is part of transformer architecture, which is that the neural network has a very limited working memory.

And if you ask a question that requires too many steps to calculate the end results, it doesn't have the working memory to store the result. Therefore it makes one up. But if you give it the working memory, which is to ask for a longer answer, the longer answer stores, the intermediate steps, therefore giving you the correct results. Talk about implementation detail, right? It's yeah. It's leaking implementation detail. It's not great. And that's why a lot of these sort of the eventual, the, right? Yeah, it's leaking implementation detail. It's not great. And that's why a lot of the eventual... I think I quoted Andrej Karpathy, who was head of AI at Tesla,

and now he's a YouTuber. And Sam Altman, who is the CEO of... Yeah, he quit Tesla to essentially pursue an independent creator lifestyle, and now he's a YouTuber. I did not know that. All roads lead to creator land. You know what I'm saying?

Like you'll be an expert in something for a while. And eventually you'll just eject to be like, you know, I want my own thing and create content and educate people around X. Oh, you know, you know, I,

I've had a, so, you know, I, my, my day job, I'm, you know,

I'm a head of a department now and I work with creators and some of them have very viable side hustles. And I just had this discussion yesterday of like, why do you still have a job if you're an independent creator? Like, isn't total independence great? And I had to remind him,

no, like career progression is good. Like you're exposed to new things, blah, blah, blah. But that's just me trying to talk about acquitting. I mean, I have a serious answer, but like this is not the, we're not here to talk about that right but mostly so I'll read out this quote you know so Sam Altman for CEO of OpenAI says I don't

think we'll still be doing prompt engineering in five years it's not about figuring out how to hack the prompt by adding one magic word to the end that changes everything else what will matter is the quality of ideas and the understanding that you want. So I think that is the prevailing view. And I think as people change models, they are understanding the importance of this. So when Stable Diffusion 1 came out, everyone was like, all right, we know how to do this. I'm going to build an entire business on this, blah, blah, blah. And then Stable Diffusion 2 came out and everything broke. All the prompts stopped working because they just expected a different model.

And you have to increase your negative prompting. And people are like, what is negative prompting? But these are all new techniques that arise out of, out of the model. And this is going to happen again and again, again, because you're relying on a very, very brittle foundation. And ultimately what's, what we want to get people to is computers should understand what we want. And if we haven't specified it well enough, they should be able to ask us what we want and we should be able to tell them in some capacity. And eventually it should produce something that we like. That is the ultimate alignment problem. Like when you hear, we talk about AI a lot, like you hear about this alignment problem, which is basically some amount of getting it to do what we want it to do, which is a harder problem than it sounds until you work with a programmer and try to tell them,

give them product specs and see how many different ways they can get it wrong. But yeah, this is an interesting form of the alignment problem. And it's interestingly, has a very strong tie with Neuralink as well, because the problem ultimately is the amount of bandwidth that we can transfer from our brain to your artificial brain. And right now it's prompts, but why does it have to be prompts? It could be images. That's why you have image to image in stable diffusion. And it could also be brain neural connections. So there's a lot in there. I'm just giving, I'll give you time to pick on whatever you respond to. Well, I went from, so I was super excited about prompting after talking with Simon, you know, a few months back, and I was super excited about prompting after talking with Simon a few months back.

And I was super excited about stable diffusion. And I went from giddy schoolboy who's just going to learn all the spells very quickly to aggravated end user who's like, nah, I don't want to go to this other website and copy and paste this paragraph of esoterica in order to get a result that I like. And so I wonder what, what's so exciting about the whole prompt engineering thing, like to us nerds. And I think maybe there's like a remnant of, well, I still get to have esoteric knowledge. I still get to be special somehow, you know, if I can learn this skill. But in reality, like what we're learning, I think by all these people using chat GPT, the ease of use of it, as opposed to the difficulty of getting an image out of stable diffusion 1.0, at least is quite a bit different. And it goes from aggravating and insider baseball kind of terms,

keywords, spells to plain English, explain what you want, and maybe modify that with a follow-up, which we'll get into chat GBT. We don't necessarily have to go into the depths of that right now. But I changed very quickly, even though I still thought prompt engineering was pretty rad. And then when you explained to me how Stable Diff diffusion to completely broke all the prompts, I'm like, oh yeah, this is a smell. This doesn't work. You can't just completely change the way it works

on people, you know, that doesn't scale. So like, yeah. And I think about all the businesses that have been built already, like, you know, there haven't been any huge businesses built on a stable diffusion, but GPT three GPT-3 has internal models as well. So Jasper recently raised like a 1.5 billion valuation. And then chat GPT came out basically validating Jasper. So all the people who bought stock

are probably not feeling so great right now. That's it. So I don't want to overstate my position. There are real moats to be built around around ai and i think that the best entrepreneurs are finding that regardless of all these flaws right the fact that there are flaws right now is the opportunity because so many people are scared off by it they're like ai has no modes you're just you're just a thin wrapper around open ai but the people who are real entrepreneurs figure it out and so

i think it's just a really fascinating case study in technology and entrepreneurship because here's a new piece of technology nobody knows how to use and productize and the people who figure out the playbook are the ones who win yeah are we back to this i mean it was like this years ago when big data became a thing but are we back to this whole world where, or maybe we never left, where data is the new oil, is the quote. To train these models, you have to have data. So you could be an entrepreneur, and you could be a technologist, you could be a developer, you could be an ML, you could be whatever it might take to build these things, but at some point you have to have a data set. How do you get access to these data sets? It's the oil.

You've got to have money to get these things. You got to have money to run the hardware to enable, like Jared, you were saying before the call, like there was speculation of how much it costs to, you know, run chat GPT daily. And it's just expensive, but the data is a new old thing. Like how does that play into training these models and being able to build the mode? Yeah. Yeah. Also, you know, one distinction we must make there is there is a difference between running the models, which is just inferences, which is probably a few orders of magnitude cheaper than training the models, which are essentially a one-time task. Not that many people continuously train, which is nice to have, but I don't think people actually care about that in reality.

So the training of the models ranges between, and let's just put some bounds for people. I love, I love dropping numbers in podcasts, by the way, because it helps people contextualize, right? Like you, you made an oblique reference to how much chat GPT costs, but you know, let's get real numbers. I think the guy who did an estimate said it was running at $3 million a month. Did you, I don't know if you heard any different, but that that's, I heard a different estimate that would have been more expensive, but I think yours is probably more reliable than mine. So let's just go with that. I went through his stuff and I was like, yeah, okay. This is on the high end. Uh, my, I came in between like, uh, yeah, one to three as well, but, uh, it's fine. And then for training the thing, so it's widely known or widely reported

a stable diffusion cost 600 K for a single run. People think the full thing, including R&D and stuff, was on the order of 10 million. And GPC 3 also costs something on the order of tens of millions. So I think that is the cost. But then also that is training. That is mostly GPU compute. We're not talking about data collection, which is a whole other thing. And I think basically there's a towering stack of open source contributions to this data collective pool that we have made over time. And so I think the official numbers are like 100,000 gigabytes of data that was trained for stable diffusion. And it's basically pulled from Flickr, from Wikipedia, from all the publicly available commons of photos.

And that is obviously extremely valuable because another result that came out recently that has revolutionized AI thinking is the concept of Chinchilla laws. Have you guys covered that in the show or do I need to explain that? Chinchilla laws is Mrs. Lamarck for me. Please tell. I like the idea though, whatever. It sounds cool. So please. Yeah. They just had a bunch of models and the

one that one happened to be named chinchilla. So they kind of went in it. It's got a cute name, but the main idea is that we have discovered scaling laws for machine learning, which is amazing. So in the sort of classical understanding of machine learning, you would have a point at which there's no further point to trade. You're sort of optimizing for a curve, and you get sort of like diminishing returns onto a certain point. And then that's about it. You would typically conclude that you have converged on a global optimum, and you kind of just stop there. And mostly in the last five to 10 years, the very depressing discovery is that this is a mirage. This is not a global optimum. This is a local optimum.

And this is called the double descent problem. If you kind of Google it, Wikipedia, you'll find it, which is you just throw more data at it. It levels off for a bit and then it continues improving. And that's amazing for machine learning because that basically precipitated the launch of all these large models, because essentially what it concludes is that there's essentially no limit to how good these models are, as long as you can throw enough data at it,

which means that, like you said, like data is the new oil again, but not for the old reason, which is like, well, we're going to analyze it. No, we're just going to throw it into all these neural nets and let them figure it out. Well, I think there's a competitive advantage though, if you have all the data. So like if you're the facebooks or for the google or you you know x y or z instagram even like instagram ads are so freaking relevant that yeah apple for sure but they're so yeah tick tock you know yeah gosh gosh tick tock and yeah the point is like these they

have a competitive advantage because they essentially have been collecting this data, you know, would be to analyze potentially to advertise to us more. But what about, you know, in other ways that these modes can be built? I just think like when you mentioned the entrepreneurial mind being able to like take this idea, this opportunity as, you know, this new ALNscape to say, let me build a mode around this and not just build a thin layer on top of GPT, but build my own thing on all together. I got to imagine there's a data problem at some point, right? Obviously there's a data problem at some point. So obviously, you know, the big tech companies have a huge headstart, but how do you get started collecting this data as a founder? I think the story of MidJourney is actually super interesting. So between MidJourney, Stability AI and OpenAI, as of August, who do you think was making the most money? I'll give you the answer. It was MidJourney. I was going to guess that. You can't just give us the answer.

It's not obvious, right? Like the closed source one that is not the big name, that is not the, doesn't have all the industry partnerships, doesn have the celebrity ceo right that's the one that had to make the most money but they launched with the business model immediately didn't they they had a subscription like out of the box yeah they did but also something that they've been doing from the get-go is that you can only access mid journey through discord why is that right because it's social or i don't know what do you think that was my guess like because they're right in front of everybody else data data oh please tell us more sean because the way that you experience mid journey is you put in a prompt it gives you four

images and you pick the ones that you like for enhancing so the process of using mid journey generates proprietary data for mid journey to improve mid journey and so from V3 to V4 of mid-journey, they improve so much that they have carved out a permanent space for their kind of visual AI-driven art that is so much better than everyone else because they have data that no one else has. That's really cool. And that's relevance or is it like quality taste?

What is the data they actually get? Preference, right? What's good? Yeah, like literally you type in a prompt, unstructuredly, it tells you, they give you four low res images

and you have to pick one of the four to upscale it. By picking that four, they now have the data that says, okay, out of these four, here's what a human picks.

And it's proprietary to them and they pay nothing for it because it's on Discord. It's amazing. That is awesome. They didn't build a ui they just they just use discord i don't know if discord knows this or cares uh but it's it's it's pretty it's pretty freaking phenomenal because now they have this it's the ultimate and scrappy right it's

like by any means necessary that's the ultimate by any means necessary right you know you'll make a beat however you can to put up the track and, you know, become the star. Right. That's amazing. That's really cool. So, you know, just to close this out, the thing I was saying about Chinchilla was more data is good. We found a double descent problem. Now let's go get all the data that's possible. I should make a mention about the open source data attempts. So people understand the importance of data. And basically, Luther AI is kind of the only organization out there that is collecting data that anyone can use to train anything. So they have two large collections of data called the stack and the pile, I think is

what it's called. Basically, the largest collection of open source, permissively licensed text for you to train whatever language models you want. And then the similar thing for code. And then they're training their open source equivalents of GPT-3 and Copilot and what have you. But I think those are very, very important steps to have. And basically, researchers have maxed out the available data.

And part of why OpenAI Whisper is so important for OpenAI is that it's unlocking sources of text that are not presently available in the available training data. We've basically exhausted, we're data constrained in terms of our ability to improve our models. So the largest source of untranscribed text is essentially on YouTube. And there's a predominant or prevailing theory that the primary purpose of Whisper is to transcribe all video, to get text, to train the models. Because we are so limited on data. Yeah. We've helped them already with our podcast.

Not that it mattered, but we've been transcribing our podcast for a while. So we just gave them a leg up. You did, you did. And that's open source on GitHub too. They probably, I mean, like ChatGPT knows about ChangeLog. They know that Jared, I don't know if I told you this yet, but I prompted, I said, complete the sentence. Who's the hosts of the ChangeLog podcast?

Well, that's the dynamic duo, Jared Santo and Adam Stachowiak. I mean, it knows who we are. I mean, maybe it's our transcripts. I don't know, but it knows. Please tell me it called us the dynamic duo. I promise you. It said that? I promise you.

It said the dynamic duo. Oh, shucks. It actually reversed the order. It said Adam Stachowiak first and then Jared Santo because usually my name is, I guess, is first because I have no clue why it's ever been that way. But it said the dynamic duo, Adam Stachowiak and Jared Santa. That's hilarious.

Hosts of the Change Low podcast. It already understands flattery. Yeah, it does. Well, I said, actually, the first prompt didn't include us. And I said, make it better and include the hosts. And that's all I said was make it better and include the hosts. I mean, in terms of reprompting or refining the response that you get from the prompt that to me is like the ultimate human way to like conjure the next available thing which is try again or do it better but give me the host too

and the next one was flattery and actually our names in the thing like so it's it's crazy anyways yeah so that is the big unlock that chat gbt enabled totally which is why usually i take a few weeks for my takes to marinate to do for me to do research and then for for me to write something but i had to write something immediately after chat gbt to tell people how important this thing is it is the first real chat ai which means that you get to give human feedback and this theme of reinforcement learning through human feedback is the low-res version of it was mid-journey. Actually, the lowest-res version of it was TikTok

because every swipe is human feedback. And being able to incorporate that into your... And same for Google. Every link click is human feedback. But the ability to incorporate that and to improve the recommendations and the generations is essentially your competitive advantage. And being able to build that as part of your UI,

which is why, by the way, I have been making the case that front-end engineers should take this extremely seriously because guess who's very good at making UI? Yeah, for sure. But yeah, ChatGPZ3 turns it from a one-off, zero-shot experience where you prop the thing and then you get the result and it's good or bad. That's about the end of the story.

Now it's an interactive conversation between you and the bot. And like, you can shape it to whatever you want, which is a whole different experience. This episode is brought to you by Influx Data, the makers of the InfluxDB time series platform. With its data collectors and scripting languages, a common API across the entire platform, and highly performant time series engine and storage, InfluxDB makes it easy to build once and deploy across multiple products and environments.

The new InfluxDB storage engine allows developers to build real-time applications even faster and with less code, faster write and query performance with a new purpose-built columnar time series database that combines a hot, compressed, in-memory data store and a cold object store. unlimited cardinality lets you slice and dice on any dimension without sacrificing performance and more options than ever to query data including native SQL support if you want to learn more and see how the new InfluxDB engine works

sign up for the InfluxDB beta program at InfluxDB.com slash changelog again InfluxDB.com slash changelog.

Complete the Sentence has been a hack for me to use, particularly with ChatGBT. I like Complete the Sentence is a great way to like easily say, just give me something long, given these certain constraints. Well, that's effectively what these models are, right? They're autocomplete on steroids. They are basically autocompleting with a corpus of knowledge that's massive and guessing what words are semantically should come next kind of a thing. In layman's terms, it's more complicated than that, of course, but they are basically auto completers.

Yeah. On that note, though, we have a show coming out. So we're recording this on a Friday, the same day we released the same podcast, but it's the week beforehand. So we had Christina Warren on. And so I was like, you know what? I'm going to use ChatGPT to give me a leg up. Let me let me make my intro maybe a little easier or just just spice it up a little bit. So I said, complete the sentence, this week on the channel we're talking

to Christina Warren about, and then I ended the quote, and I said, and mentioned her time at Mashable Film and Pop Culture and now being a developer advocate at GitHub. And I gotta say, most of, 50% of the intro for the episode with Christina

is thanks to ChatGPT. I don't know if I break a terms of service by doing that or Christina is thanks to chat GBT. I don't know if I break a terms of service by doing that or not, but like, do I, I don't know if I do sue me, I'm sorry, but don't sue me. Don't sue us. We'll take it down. We'll re we'll exit out, but it was mostly, yeah, we'll rewrite it. But I mean, it's basically what I would have said. So there's a nice poetry, you know, there nice poetry. There's a YouTuber who's been on this forever, Two Minute Papers. And what he often says is, what a time to be alive.

And this is very much what a time to be alive, but not just because we're seeing this evolve live, but because we get to be part of the training data. And there was a very interesting conversation between Lex Friedman and Ajay Karpathy when he was inviting him onto the show. He said, our conversation will be immortalized in the training data.

This is a form of immortality because we get to be the first humans essentially baked in. Essentially baked in. Hello, world. Like 100, 200 years from now

if someone has the ChangeLog podcast, they will keep having Jared and Adam pop up because they're like, in the goddamn training data. They're like, come on,

these guys have been dead for a long time. Let them go. Give them their RIP. Which is, which is a poetic and nice. Yeah. Yeah.

It is a good time to be alive. I think it is interesting too. I just wonder, I mean, this might be, you know, jumping the shark a little bit, but I often wonder at what point does humanity stop creating?

And at some point, a hundred years from now, maybe more, I don't know. We're going to be maybe sooner given how fast this is advancing that we'll create only through what was already created. At what point does it, is the snake eating the snake kind of thing?

Like, is there an end to human creativity at some point because we are just so reliant at some point shape or form on contra results because of training data and this just kind of like morphing to something much much bigger in the future also i have a optimistic attitude to that this question basically is asking can we exhaust infinity and so my obvious answer is no there's a there's a more concrete stat i can give you which is i think this is floating around out there don't call me on the exact number

but apparently 10 of all google searches every single year have never been asked before for and google's been around for like 20 years that's a big percentage it's still true so it's it's on it might be like seven it might be 13 well is it trending down though? is it trending down? is it 10% per year but is it like trending down to like 8? is it because we put the year in our searches?

yeah it's true Jerry good one but anyway so that's what the SEO people talk about when they talk about long tail like you always want

there's just the amount of infinity is always is bigger than our capability of creating to fill it. I feel like if you look at us in an abstract way, humans, we are basically taking in inputs and then generating outputs.

That's creativity, right? So I think what we're just doing is adding more to the inputs. Now we have computers that also take in inputs and generate outputs. Everything's already a remix, isn't it? Our life experience and everything that goes into us and then something else produces a brand new thing, which isn't really new, but it's a remix of something else that we experienced. I feel like we're just going to keep doing that

and we'll have computer aided doing that and the computer eventually maybe will just do the actual outputting part, but we somehow instruct it. I don't think there's going to be an end to human creativity as the AI gets more and more output. What's the word? When you're just not notorious. What's it called? When you just can't stop outputting stuff. I don't know.

Prolific. Prolific. As the AI gets more and more output prolific and overwhelms us with output, I think we're still going to be doing our thing. It's the ultimate reduction in latency to new input. Think of 100 years ago, creative folks were few and far between.

You know, they had miles between them, you know, depending on your system. Maybe it's kilometers. No offense. But, you know, there's distance of some sort of magnitude and the lack of connection and shared ideas. So that's the latency, right? And now, you know, it's the latency to the next input is just so small in comparison. And we'll get reduced to basically nothing. So we'll just constantly be inputting and outputting creativity.

We'll just become like a creative exhaust system with zero latency, nonstop creativity. Go, go, go. Well, so I think this is where you start. I don't know about you, but I feel a little bit uncomfortable with that, right? Like our entropy is always increasing in the universe. You know, we're contributing to increasing noise and not signal. And that is a primary flaw of all these language models, which is they are very confidently incorrect. They have no sense of physics, no sense of logic. They will confidently assert things that are not true. And as long as they're trained on

sounding plausible rather than being true. Right. And they're kind of like me when I was in college, you know? Exactly. Just so much confidence, but wrong most of the time. Exactly. Which happens to Galactica, which is this sort of science LLM from Meta,

where there were Jan LeCun, who was like one of the big names in tech, was like this this thing would generate papers for you and in within three days the internet tore it apart and they had to take it down it was a very very dramatic failure of this kind of tech because you're talking about biology and science and medicine and you can't just make stuff up like that right so like in the world in the world where chat gpt operates today which is really in the world of fiction and kind of BSing for lack of a better term, like writing intros to a podcast, you know, like it doesn't have to be correct necessarily. It can be like close enough to correct.

You can massage it. Of course, you can cherry pick to get the one that you like. But when the rubber hits the road, like on serious things like science, right. Or, you know, how many of these pills do I need to take? I guess that is also, that's health science. So science and other things, it's like, it can't be correct 60% of the time or 80 or even like 95. Like there's like a, it's got to reach that point where you actually can trust it.

And because we're feeding it all kinds of information that's not correct, de facto, like how much of the internet's wrong, most of it, right? I mean, medicine, though, has evolved too, and it hasn't always been correct, though it's also very serious. You get advice from a doctor 10, 15 years ago, they say with full confidence and full accuracy, but it's only based on that current data set. But you can sue them for malpractice and stuff, right?

How do we take recourse against? You can't if they actually have malpractice. They can be wrong because it's as much science as possible to make the most educated guess. It's malpractice when there's negligence. It's not malpractice when they're wrong. A good doctor will actually go up to the fringe and say, you know what? I'm not 100% sure about this. It's beyond my knowledge. Here's what you can do. Here's the risks of doing that. Whereas the chat bots,

the chat GPT thing is like, the answer is seven, you know? And you're like, actually it was 12. It's like, oh shoot. Well, I think when there's mortality involved, maybe we, you know, there's going to be a timeframe when we actually begin to trust the future med GPT, for example. I don't know if that's a thing in the future, but something that gives you medical results or responses based upon data, real data potentially that you get there, but it's not today. Well, I think this goes back to the data point that you made. And I think where we go from like the 95, let's just make it up numbers here, but like 95% accuracy to get it to like 98 and a half or 99%, like that's going to require niche, high value, high signal data that may be like this medical facility has because they've been collecting it for all these years and they're the only ones who have it. And so maybe

that's where you like carve out proprietary data sets that take these models from baseline of accuracy to like, in this particular context of health, it's this much accuracy. And then maybe eventually you combine all those and have a super model. I don't know, Swix, what do you think?

I love the term super model. I think the term of art in the industry is ensemble, but that just multiplies the cost, right? Like if you want to run a bank of five models and pick the best one, that obviously is six X's your cost. So like not super interesting, good for academic papers,

but not super interesting in practice, because it's so expensive. Oh man, there's so many places to go with this stuff. Okay, there's one law that I love, which is Brandolini's law. I have this tracking list of eponymous laws. Brandolini's law is people's ability to create bulls**t far outseeds the ability of people to refute it.

Basically, if all of this results of this AI stuff is that we create better engines, it's not great. And what you're talking about, the stuff with the 90% correct, 95% correct, that is actually a topic of discussion. It's pretty interesting to have the SRE-type conversation of how many nines do you need for your use case and where are we at right now?

Because the number of nines will actually improve. We are working on, you know, sorry, we as in the collective human, we, not me personally. The royal we, yes. The royal we. Like humanity is working on ways to improve, to get that up. It's not that great right now.

So that's why it's good for creativity and not so much for precision, but it will get better. One of the most viral posts on Hacker News is something that you featured, which is the ability to simulate virtual machines instead of ChatGPG3, where people literally open, I mean, I don't know how crazy you have to be, but open up ChatGPG3, type in LS, and it gives you a file system. But that only exists, it's not a real file system. It's just one that's inside. It's not a real file system for now. It's not a real file system. It's just one that's inside. It's not a real file system for now. It's not a real file system for now

because it hallucinates some things. If you ask it for a git hash, it's going to make up a git hash that's not real because you can verify it with MD5. But how long before it learns MD5? And how long before it really has a virtual machine inside of the language model? And if you go that far,

what makes you so confident that we're not in one right now? So now I'm uncomfortable. That actually is a very short hop into the simulation hypothesis because we are effectively simulating a brain. And if you get good enough

at simulating brains, what else can you simulate? What else would you want to simulate? I mean, that's the holy grail, a brain. I'm so like, yeah, I really like Ahmad. So Ahmad Mostak is the CEO of Stability AI. He's like, we're completely unconcerned with the AGI.

We don't know when it'll get here. We're not working on it. But what we're concerned about is the ability to augment human capability, right? People who can't draw now can draw. People who can't write marketing text or whatever can now do that. And I think that's a really good way to approach this, which is we don't know what the distant future is going to hold, but in the near future, this can help a lot of people.

It's the ultimate tool in equality, right? I mean, if you can do... Yeah, there's a super interesting use case. So there's a guy who is like sort of high school educated, not very professional, applying for a job. And what he used ChatGPT to do was like, here's what I want to say. And please reword this in a professional email. And it basically helped to pass the professional class status check. Do you know about this status checks? Like all the sort

of informal checks that people have, like, oh, like, yeah, we'll fly you in for your job interview. Just, you know, put the hotel on your credit card. Some people don't have credit cards. And likewise, when people email you, you judge them by their email, even though they just some, so they haven't been trained to email, to write professionally. Right. And so yeah, GPT is helping people like that. And it's a huge enabler for, for those people. Huh? That is, I mean, I like that idea, honestly. I mean, cause it does able more people who are less able. It's a net positive. Yeah. I mean, I like that idea, honestly. I mean, because it does able more people who are less able. It's a net positive. Yeah, I mean, I have, you know, I seem generally capable,

but also I have RSI in my fingers and sometimes I can't type. And so what this, what Whisper is enabling me to do and Copilot, so GitHub at their recent GitHub universe recently announced a voice enabled Copilot. And it is good enough for me to navigate VS code and type code with Copilot and you know, voice transcription. Those are the two things that you need. And they are now actually good enough that I don't have to learn a DSL for voice coding like you would with Talon or all the, all the prior solutions.

You know, it's the ultimate, if you're creative enough, it's almost back to the quote that Sam had said that you liked. Well, I'm going to try and go back to it. He says at the end, because they were just able to articulate it with a creative eye that I don't have. So that to me is like, you know, insight, creativity. It's not skill, right? It's the ability to dream, which is the ultimate human skill, which is since the beginning of time, we've been dreamers. This is a new brush.

And some artists are learning to draw with it. There will be new kinds of artists created. Provided that people keep making the brush, though. It's a new brush. But the secret's out. The secret's out that you can make these brushes. Right. Yeah, but you still have to have the motivation to maintain the brush, though.

What about access, too? Right now, you're talking about somebody who's made able and isn't otherwise with, let's just say, ChatGPT, which is free for now. But OpenAI is a for-profit entity and they can't continue to burn money forever. They're going to have to turn on some sort of a money-making machine and that's going to inevitably lock some people out of it. So now all of a sudden, access becomes part of the class, doesn't it?

You can afford an AI, and this person cannot. So that's going to suck. It seems like open source could be for the win there, but like you said, Swix, there's not much moving and shaking in that world. Well, I haven't stopped thinking about what Swix said last time we talked, which was above or below the API, which is almost the same side of the coin that we talked about last time, which is like, this is the same thing. Yeah, well, Chad, GBT is an API, isn't it?

Nice little callback, nice. I really haven't been able to stop thinking about that. Every time I use any sort of online service to get somebody to do something for me that I don't want to do because I don't have the time for it, or I'd rather trade dollars for my time, I keep thinking about that above or below the API, which is what we talked about. And that's what Jerry just brought up. It's the same exact thing.

One more thing I wanted to offer, which is the logical conclusion to generative. So that post where we talked about why prompt engineering is overrated, the second part of it is why you shouldn't think about this as generative. Because right now, the discussion that we just had was only thinking about it as a generative type of use case. But really, what people want to focus on going forward is, well, two things. One is the ability for it to summarize and understand and reason. And two, for it to perform actions.

So this is what the emerging focus is on agentic AI, AI agents that can perform actions on your behalf, essentially hooking it up to giving it legs and arms and asking it to do stuff autonomously. So I think that's super interesting to me because then you get to have it both ways. You get AI to expand bullet points into pros and then to take pros into bullet points. And there's a very funny tweet from Josh Broder, who is CEO of Do Not Pay, which is kind of like a... Yeah, I'm a fan of him.

Yeah, fantastic, right? So what Do Not Pay does is they get rid of annoying payment UX, right? Like sometimes it was parking tickets, but now they're trying to sort of broaden out into different things. And so he recently tweeted that Do NotPay is working on a way to talk to Comcast to negotiate your cable build down. And since Comcast themselves are going to

have a chatbot as well, it's going to be chatbots talking to each other to resolve this. Oh man. It's like a scene out of Futurama or something. Yeah. So I'm very excited about the summarization aspect, right? One of the more interesting projects that came out of this recent wave was Explain Paper, which is you can throw any academic paper at it and it explains the paper to you in approachable language and you can sort of query it back and forth. I think those are super interesting because that starts to reverse

Brandolini's law. Instead of generating bullshit, you're taking it and getting it into some kind of order. And that's very exciting. Yeah. 17 steps back, it makes me think about when I talk to my watch and I say, text my wife. And I think about who is using this to their betterment. And I'm thinking, we're only talking about adults for the most part. My kid, my son Eli, he talks to Siri as if she knows everything, right? But here's me using my watch to say, text my wife, I say it, it puts it into the phone, and the last thing it does for me, which I think is super interesting for the future,

is this AI assistant is send it, is the final prompt back to me as the human should I send this and if I say no Siri doesn't send it but if I say send it guess what she does she sends it sends it she I love this idea of the future of like maybe some sort of you know smarter AI assistant like that I mean to me that's a dream I'd love that yeah I was watching this clip of the first Iron Man when you know robert darny jr is kind of working with his bot to work on his first suit and he's just talking to the bot like here's what i want you to do sometimes he gets it wrong he slaps it on the head but like more often than not he gets it right and this is why like i've been um you know west boss recently

twitter like should we like this is actually really scary should we be afraid as engineers like this is going to come for our jobs and i I'm like, no, like all of us just got a personal junior developer that should excite you. Yeah. And it seems like it's particularly good at software development answers. You think it's because there's lots of available text? I mean, think about like things that it's good at. It seems like it knows a lot about programming. I have a list. Do you want a list yeah so writing tutorials it's very good literally uh table of content section by section explaining like you should first you should npm install then you should do x then you should do y debugging code like explaining like just paste in your error and

paste in the source code and it tells you what what's wrong with it dynamic programming it does really well translating dsls i think there will be a thousand DSLs blooming because the barrier to adoption of a DSL has just disappeared. So why would you not write a DSL? No one needs to learn your DSL. What is this, CodePilot you're using or ChatGPT that you're- ChatGPT3. I have a bunch of examples here I can drop in the show notes. AWS IAM policies. Hey, I want to do X and Y in AWS guess what there's tons of documentation AGPT knows AWS IM policies

code that combines multiple cloud services this one comes from Corey Quinn 90% of our jobs is hooking up one service to another service you can just tell it what to do

and it just does it right there's a guy who like I fed my you know my college computer network's homework to it and it gave the right result

which is pretty interesting refactoring code from Elixir to PHPp is another one that has been uh has been done uh and obviously advent of code which is when we're recording this in december now the person who won so evident code for the first hundred people is a race whoever submits the correct answer first wins it and the number one place in in thevalon code this year was a chat GPT guy. So it broke homework. Like this thing has broken homework in an interview and take home interviews, basically. Completely. So nice though. Like I've only used it a little bit while coding, but it's got, it's two for two. I'm just like drilling my exact questions and like, just stuff like, uh, how do you match any character that is not an at in a regular expression?

Like that's. Oh yeah. Explaining regexes. Yeah. Like that was my question. Like I know exactly what I want, but I don't, I can't remember which is the character.

And so I just asked it and it gave me the exact correct answer and an example and explained it in more detail. If I wanted to go ahead and read it and it warned me, Hey, this is not the best way to test against email addresses, but you know, here it is. So I was like, all right, this is a, this is a good thing for developers for sure. But you can't trust it. So you have a responsibility as well. You can't just say, you can't write bad code, have something bad happen and go,

Oh, it wasn't my fault. It was chat to BC. Well, you can't paste sack overflow answers into your code either. You have the responsibility. Exactly. Yeah. Yeah. I mean, you can, but you're going to get fired, right? Like if the buck stops at you, not at the Stack Overflow answer person, you can't go find them and be like, why were you wrong? Right. It stops at you. Yeah. So, you know, I think the way I phrased it was, you know, do you know about this trade offer meme that is going around? So it's trade offer. You receive better debugging, code explanation, install instructions, better documentation,

elimination of your breaking of flow from copying and pasting in Stack Overflow. You receive all of these benefits in exchange for more code review. There is a cost, which is code review. You have to review the code that your junior programmer just gave you. But hey, that's better and easier

than writing code yourself. Yeah, because you got a free junior programmer working for you now. There's a guy that says, I haven't done a single Google search or consulted any external documentation for the past two days. And I was able to progress faster than I ever had

when learning a new thing. I mean, it's just, it's amazing. And Google should be worried. Yeah, that's what I say. Is this an immediate threat to Google? Now, I did see a commenter on Hacker News, so I'm not sure if you saw this one, from inside of Google,

talking about the cost of integration. Yes, yeah, I've read basically every thread. It's just a full-time job, but this is so important. I don't do this for most things, right? I think this is big enough that I had to drop everything and go read up on,

and, you know, not be an overnight expert, but like at least try to be informed. And that's all I'm doing here, really. But yeah, yeah. You want to read it out? Yeah, yeah, yeah. So in summary,

they were responding, this is on a thread about ChatGPT, and they say, this is a Googler, and they say, it's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day when you take into account serving costs, added latency, and the fact that average revenue on something like a Google search is close to infinitesimal, which is a word I can't say out loud, already. I think I remember the presenter saying something like they'd want to reduce the cost by at least 10 times before it could be feasible to integrate models like this in products

like Google Search. A 10x or even 100x improvement is obviously an unattainable target in the next few years. So I think technology like this is coming in the next few years. So that's one insider's take on where Google stands. Obviously, Google has tons of resources dedicated to these areas of expertise, right? It's not like Google is asleep at the wheel and is going to completely have their lunch eaten by OpenAI. But right now, there's a lot of people who are training new habits right like they're like i'm not going to use google anymore i'm going to start using open ai i think it's something on the

order of 1 million users in their first few days have signed up how long can google potentially bleed people before it becomes an actual problem i don't know i don't know the answer to these things so uh there's a there's one way in which you can evaluate for yourself right now. And I think that's the most helpful, constructive piece of advice that we can give on this podcast, which is, you know, we're covering something that is moving very live,

very fast. Everything that we say could be invalidated tomorrow by something new. But you could just run ChatGPT3 alongside of all your Google searches. That's a very, very simple way to evaluate would this replace Google for you? Just run it twice, every single time. And so there's your Google searches. That's a very, very simple way to evaluate would this replace Google for you? Just run it twice, every single time.

And so there's a Google extension. I'll link it. It's a Wong2 ChatGPT Google extension. I'll put it in the show notes. And yeah, I have it running. It's not that great, surprisingly. So ChatGPT is optimized for answering questions. And sometimes I don't put questions in there.

I just put the thing I'm looking for and Google's pretty good at that, it turns out. Right. See, because you are a expert level Google prompt engineer, right? Like you know how to talk to Google. We have optimized to Google prompting, yes. I know, like if I need to search for

within a certain date range, I know how to do that in Google. I can't do that in ChatGPT3. If I need to look for PDFs, I know how to do that uh if i want to look for reddit and constrain the the site to reddit i know how to do that uh chat gpt3 has no concept of attribution no concept of uh date ranges and stuff like that right but yeah it is just like better in some things and worse than other things and that is the nature of all new technology right it just has to be better at one thing that it cannot get you cannot get anywhere else and it has a permanent hold in your mind whenever you need

that thing done you will turn to chat gp3 or any any other new technology right like i love this like sort of meta philosophy about technology adoption like because all new toys just generally are worse than the things that they replace except in one area and that's the area needs to matter and if it does matter it will win because they will fix the they'll fix the bugs yeah oftentimes than the things that they replace, except in one area. And that area needs to matter. And if it does matter, it will win because they'll fix the bugs. Yeah, oftentimes with disruption, that area is cost, like acquisition cost. Sometimes it's convenience.

And maybe I guess sometimes it's accuracy. There's different metrics, but it's gotta be the one that matters. If it's like a little bit, if it's marginally better at things that don't matter, you're not gonna disrupt. But if it's marginally better at things that don't matter, you're not going to disrupt. But if it's a lot better at one thing that matters a lot, even if everything else sucks, you'll use that thing. Yeah, exactly.

So it's interesting because, you know, Google has a few things going for it. By the way, it has one of the largest training repositories of text that no one else has, which is Gmail. But the most impressive thing is being able to ship a Gmail is the little autocomplete, like looks good. Okay. You know, type the little buttons that you see in the, in the smart replies. You guys ever use those? Do you ever click on those?

I use that. I use that. Say some typing. Yeah. When I used to actually use Gmail directly to compose my emails or respond, you know, I would tab to complete all the time. If the response was like, yeah, I was going to say that emails or respond, I would tab to complete all the time if the response was like,

yeah, I was going to say that. There's a billion little ways that AI is built into Google right now that we just don't, we take for granted because we don't feel it because there's no prompts. We need a prompt.

Even if OpenAI did eat Google's lunch, Google would just acquire it or you know, or something. You would think so. Yeah, but I would say that probably OpenAI is not for sale. Like they have this world conquering ambition that would just not let them settle for anything less than global domination, which is a little bit scary, right?

Yeah, I think they're probably going the distance as they're playing, it seems like. Well, if anything, Microsoft should have bought them when they had the chance, uh, because that was Bing's opportunity. And, uh, I don't think, I don't think that ever came to pass probably because Sam Altman's smart enough not to do that deal, but yeah. So like Google's probably so like, okay, let's take that line of thinking to the logical conclusion. What would you feel if Google started auto-completing your entire email for you and not just like individual, like two or three words, you would feel different. You would feel creeped out. So Google doesn't have the permission

to innovate. I wouldn't freak out if I opted in though. Like if I was like, this technology exists and it's helpful, I'll use that. Now, if it just suddenly started doing it, yeah, I creeped out. But if I'm like, yeah, this is kind of cool. I opt into this enhanced AI or this enhanced auto-completion or whatever, you know, simplifies the usage of it or whatever. Yeah. So there's actually some people working on an email client that does that for you. So Evan Conrad is working on every prompt email, which is essentially you type a bunch of things that you want to say, and you sort of batch answer all your emails with custom generated responses from GPT-3. It's a really smart application of this text-to-email

that I've seen. But I just think like you would opt in. The vast majority of people never opt into anything. No, yeah, most people don't opt in. That's just not the default experience. So I'm just saying like one reason that Google doesn't do it is yeah, we're just too big, right?

That is essentially the response that you read out from that engineer. Like we are like this doesn't work at Google scale. We can't afford it. It will be too slow. Like whatever. That's kind of a cop out, I feel like, because Google like should be capable. These are the best engineers in the world. They should be able to do it.

Well, he does say he thinks it's coming in the next few years. So, you know, he's not saying it's impossible. He's saying they're not there yet. And I will say I'm giving ChatGPT the benefit of my wait time that I do not afford to Google. I do not wait for Google to respond. I will give ChatGPT three to five seconds. Yeah. Because I know it's like a new thing that everyone's hitting hard.

Right. But like if they just plug that in, it would be too slow. Like I wouldn't wait three to five seconds for a Google search. By the way, that's a fascinating cloud story that you guys got to have on. Find the engineer at OpenAI that scaled ChatGPT-3 in one week from zero to one million users. Yeah, totally. Well, if you're listening or you know the person, this is an open invite. We'd love to have that conversation.

Yeah, I know that I've seen the profile of the guy that claimed to watch it live so that he would know, but I don't know who would be responsible for that. That is one of the most interesting cloud stories probably of the year. And Azure should be all over this. Azure should be going like, look, they handled it, no problem.

This is the most successful consumer product of all time. Come at us, right? That's true. They should. They're the number three cloud right now. They just like, they're one thing. They just have time to shine. Like they got to do it, you know? And does anybody even know that Azure is behind OpenAI?

I'm sure you can find out, but like, is that well known? I didn't know that. Oh, it's very public. Microsoft invested a billion dollars in OpenAI. Okay. Did you know that, Adam? So I'm trying to gauge the public knowledge. What we didn't know was that it was at a valuation of $20 billion.

So openly, I went from this kind of weird research lab type thing into one of the most highly valued startups in the world. You think Microsoft got their money worth? I think so. It's a wash right now because they probably cut them a lot of favorable deals for training and stuff. So it's more about being associated with one of the top AI names. This is the play that Microsoft has been doing for a long time. So it's finally paying off.

So I'm actually pretty happy for them. But then you have to convert into getting people who are not named OpenAI onto this thing. This episode is brought to you by our friends at Retool. Retool helps teams focus on product development and customer value, not building and maintaining internal tools. It's a low-code platform built specifically for developers. No more UI libraries, no more hacking together data sources, and no more worrying about access controls. Start shipping internal apps to move your business forward in minutes with basically zero uptime,

reliability, or maintenance burden on your team. Some of the best teams out there trust Retool, Brex, Coinbase, Plaid,

DoorDash, Legal Genius, Amazon, Allbirds, Peloton, and so many more. The developers at these teams trust Retool as their platform

to build their internal tools and that means you can too. It's free to try, so head to retool.com slash changelog. Again, retool.com slash changelog. what's the long-term play here though i mean like if microsoft invests that kind of money and we're using chat gpt right now we're willing to give it extra seconds potentially even a minute if the answer is that important to you that you wouldn't afford to Google. What's the play for them? Will they turn this into a product? How do you make billions from this? Do you eventually just get absorbed by the fangs of the world and next thing you know, now this incredible future asset to humanity is now owned by essentially folks we try to host our own services for. We're hosting NextCloud locally so we can get off the Google Drives and whatnot.

And all this sort of anti-whatever. I mean, what's the endgame here? God. Am I supposed to answer that? Do you have an answer for that? Do you have an answer? I mean, that's what I think about. Let's ask ChatGPT what the endgame is.

No, I mean, short term doesn't seem like open AI becomes the API layer for every AI startup that's going to start in the next five or ten years. Aren't they just charging their fees to everybody who wants to integrate AI into their products, pretty much? That's not endgame, but that's short-term business model, right? That is a short-term business model, yeah. I bet they have much more up their sleeves. I don't actually know. But they did just hire their first developer advocate,

which is interesting because I think you'll see, you'll start to hear a lot more from them. Well, there's two things I will offer for you. So one is, it's a very common view or perception that AI is a centralizing force, right? Which is Adam, what you were talking about, which is, does this just mean that the big always get bigger, right? Because the big have, have the scale and size and data advantage. And one of the more interesting blog posts, I'm sorry, I can't remember who I read this from, was that actually one of the lessons from this year is that it's not necessarily true because AI might be a more decentralized force because it's more amenable to open source and crypto instead of being decentralized turned out to be more centralized than people thought so like the the two directions of centralized

versus decentralized the common perception is that yeah it's very centralized very centralized and crypto is very decentralized the reality was that it's actually the opposite which is uh which is fascinating to me as a thesis right like is that the end game that ai eventually gets more decentralized because people want this so badly that there are enough researchers who go to NeurIPS to present their research papers and tweet out all this stuff that diffuses these techniques all over the place. And we're seeing that happen, helped in large part by stability AI. Like the proof that stability as an independent, like an outsider company, like not a ton of connections in the AI field, did this humongous achievement,

I think it's just a remarkable encouragement that anyone could do it. And that's a really encouraging thing for those people who are not fang and trying to make some headroom in this world. So that's one way to think about the future. The second way to think about like, whether or not who monetizes

and who makes the billion dollars on this, there's a very influential post that I was introduced to recently from Union Square Ventures called the myth of the infrastructure phase, which is directly tackling this concept that everyone says, when you have a gold rush,

sell picks and shovels, right? And it's a very common thing. And presumably AI being a gold rush right now, you should sell picks and shovels, which is you should build AI infrastructure companies. But really, there are tons of AI infrastructure companies right now. There are a dime a dozen. Really, they're all looking for use cases. And basically, the argument of the myth of the infrastructure phase is that technology swings back and forth between app-constrained and infra-constrained. And right now, we are not

infrastructure-constrained, we're app-constrained constrained. And really it's the builders of AI enabled products like TikTok that know what to do with the AI infrastructure that can win. So I'll introduce those concepts to you. I don't know what specifically that means. I'm looking out for opportunities like that myself. But I think it's apps. I think it may not be infra because it's going to trend very very hard towards commodity infrastructure provisioning and i don't really care where i get my a100s from a100 being the predominant nvidia chip that is being used to train all these models which by the way costs two hundred thousand dollars per chip

which is insane imagining a gpu costing that much yeah i also think about you know something you know kind of one-step removed to some degree you know, is the future innovation that changes the direction of humanity tied to capitalism? Right. Because like the the innovation that happens in this space in particular, which we see is very beneficial, you know, abling those who are not able to is a very beneficial thing, obviously. But if it's tied to the fact that it's tied to a company that can profit, it's, you know, I'm not anti-capitalistic by any means. And I'm also not like only capitalistic. It has to make money or else it dies. But that's kind of the world we're in. As a startup, all roads lead to either, you know, acquisition, IPO, or you just have an amazing customer base that you just are profitable on your own there's some

sort of exit like there's investment there's exit and I just wonder how this advancement this innovation that totally is the future of humanity you know how it lives in a capitalistic world where it may or may not die and then eventually this thing has to profit and so therefore you know is it value or is it the fangs i guess i don't trust the fang so much like if they get their hands and they have the centralization of this thing and the other controllers like google is of search now i love google in many respects but in a lot of case i don't want to give it all my data even though it knows everything about me i mean it knows my youtube history so it knows probably the most i ever wanted to know about me just because

of that like it knows my interests just because of that. But like, I'm weary about that. Right. Like, I don't know. Call me a skeptic. Call me cynical. Well, I will paraphrase Churchill when I say, you know, capitalism is the worst form of economic organization,

except for all the others that we've tried. Right. Yeah. I believe that's what makes things win. It's just, you know, how, and do we end up getting like a version that's what makes things win. It's just how. Do we end up getting a version that's sellable versus a version that's usable? Maybe that's the same, but maybe it's not. Well, if this new thesis is correct,

it's not actually a centralizing force. Perhaps the open source side of things will step up to the plate. It's so valuable that other people can do it in the small. Maybe we'll learn new techniques where you don't have to have all the world's data to get good results. I mean, we don't know exactly how this thing is going to play out, but it's possible that we will see

even large organizations like OpenAI is going to become has an open source whisper thing, which we can all run on our machines. Swix, they have a devious reason for it to open source that sucker. Devious. A capitalistic reason for open sourcing that sucker.

Yeah, it's not devious. Well, it's a bit underhanded. If it's pitched, it's altruistic. And then it's like, what? We're doing it to get this stuff transcribed. They never said altruistic. They didn't even say the intention. They just released it. That's all.

Well, it's just the de facto open source stories. Altruism, isn't it? Like, that's just what gets assumed. Yeah. Yeah.

It's like, why would you do this? It's like, well, we're not going to tell you. So you assume it's altruistic. Anyways, regardless of their intentions,

my point is that we do have open source things that are happening and perhaps they will continue to thrive and we'll have alternatives as we have had historically. Well, okay. We have to figure out licensing. This is a huge point of discussion because the code, like MIT licensing the code doesn't matter. It's the data that needs licensing and the model weights that needs licensing. And we don't have a legal framework for that. So OpenRail is the current form. But even there's been like five different variations of OpenRail right now.

And there's a lot of back and forth about like what responsible AI open sourcing is. And it's super interesting to follow along. If you're interested, look up for Yannick Kilcher, who has been, I think, one of the best sources of getting up to speed on AI on YouTube that I've found. He's kind of like a weird personality, but he made a real impact because he made his own license and Stability adopted it for Stability V2. Which is huge. A random YouTuber is just making up his own license and Stability going like, yeah, that looks good. Wow.

It's fantastic. And then, of course, all the open source, you know, original is the fundamentalist going like, this isn't open source. It's not OSI approved. I'm like, OSI is relevant now because the information, the value of the work just moved from the code layer to the weights and data layer. Right. And while you're giving resources, I have one as well. Our friend Louis Villa from Tidelift, tech lawyer Louis Villa, has a newsletter called Open-ish Machine Learning News. Of course, he's very much on the licensing side of being a lawyer. And he's following all of the things and summarizing and writing his thoughts. I've been following his newsletter for six weeks now, and I've been really enjoying it if you want to follow more things check out his as well swix let's talk about your sub stack man let's get some subs to your sub stack as we're tail out latent space uh you know prompt i'm prompting you for

the promo latent space diaries yeah so it's lspace.swix.io the reason it doesn't have a domain like i could have got swix.ai or something, is because I wasn't sure when I started this how much of a fad it was. And I was like, oh, you know, I don't know if I should put this on my regular blog because this could come and go. I'm not an AI expert in any way.

But this then just became so dominant in my thinking that I had to have an outlet. So I started this substack. And yeah, lspace.swix.io, It's got all the perspectives that I've been collecting. I just got a submission for my first guest post. So we'll see if it might become more of a thing, but right now it's just my thinking long and I have a GitHub repo with all my sources. So I tend to keep the blog and the sources separate because the sources evolve much faster than the blog. So highly, you're welcome to pick and choose from whatever you wish.

But thanks for having me on and letting me plug the newsletter. We appreciate your thinking. We appreciate you, you know, in the public doing this all the time, all the things. It's actually kind of scary. I appreciate your courage then. You're the expert now, Swix. We turn to you for the expert opinions. It's true.

Yeah. But no, like I do try to model it because this is the whole learning public philosophy, right? Which is you don't need to be an expert. You just need to put in the work. You need to think really hard. You need to do the research, but then put it out there and let people correct you.

And that's how you improve. And that's the only thing you can ask for. Well, you know, your career path and trajectory is proof that it can be successful if done with, I think, a humble mindset, which is what I think you approach it with. Putting your thinking out there and not feeling like you have to be correct or right, or the expert, to me is the epitome of humbleness. So I think if you take it with that approach, you'll get those kinds of results or similar results of every time I think about somebody doing it in public, it's it, you're a good example of how to do it well. Well, uh,

thank you so much. But you, you guys have been doing this way longer than I have, you know, how long has been the change lock been running? More than 13 years now. I think we just, was it just 13 years, Jared? Yeah. November 19th, we were born 2009. So do the math. 13 plus years. You guys, you're the originators of this space. I'm just a mere copy. Oh, man. Aren't we all though eventually within this world? Didn't we just talk about that? Everything's a remix. That's what I was saying earlier. It's all a remix. Yes. And even that is a remix of a guy who made a series of YouTube videos called Everything's a Remix and I'm just spitting his game

I was gonna say I was wondering if you watched that video oh yeah absolutely that's why I think about it all the time when I think about how we take stuff in

and output new things I'm like it's just remixes all the way down nothing new under the sun Jared nothing new under the sun well it's certainly reassuring to think about that too

that we're not snowflakes that we we're just remixes like literally half your mom and half your dad we're all literally remixes it's true well

that's it Swix thanks for coming on man appreciate your thinking anything else we're going to have to ask, I guess. I'm listening to the show as we're doing it, and I'm thinking, like, gosh, there's so many links to link out in the show notes.

My only request from you, Swix, as a guest, is also to be our show notes maker or helper, because you've just dropped so many links that I'm not sure I'll be able to pick up or will pick up behind the scenes. So please, as soon as we release this episode, you know, know commit back help us with missing links as necessary oh yeah please because uh i want these notes to be super rich obviously we'll point back to your repo and you know l space diaries and whatnot but a summary in the notes would be great i think

we'll catch a lot of them but i'm just asking you to pick up my gaps is all of course of course uh it'd be my my pleasure yeah i love by the way i love that you put so much effort in the notes would be great. I think we'll catch a lot of them, but I'm just asking to pick up my gaps is all. Of course, of course. It'd be my pleasure. Yeah, I love, by the way, I love that you put so much effort in show notes. Like that's another thing that podcasters don't do. But yeah, look like, I think for people who are maybe feeling a bit of FOMO, I think AI researchers are also feeling the FOMO.

This thing is, this is a progressing at enormous paces. And I think a lot of people that i talk to are like i don't have time to keep up on this and like but also at the same time like this could change my entire world like what should i be feeling and i always go like you know people always underestimate what they can do in in 10 years but then people overestimate like the amount of impact that these uh tidal waves are going on on a day-to-day basis. So basically like keep on top of the high level trends, like, you know, listen to podcasts like The Change Log and get to know that stuff, but you don't have to worry about it reaching you

right away. You have some time to keep up to speed on it. So I think that's my general message. Like, don't worry, like technology always takes longer than you expect to diffuse about it. The fact that you listen to the very end of a very long and rambly podcast means that you're on the cutting edge because there's probably 90% of the world population that has just never heard of this and doesn't care. You know, and they're still watching

the World Cup or something. I just like literally could not care less about the World Cup. But for them, that is their entire world. Wait, the World Cup's going on? Oh man. So I tell people that all the time Cup's going on? Oh, man. So I tell people that all the time, like business and tech is my sport. That's my response to folks who say, well, do you watch the game? No. What game? And I say, well, ask me who did what or

who got acquired there, how much for, what the next big trend is in technology. And I'll know it for the most part, or at least be aware that's my sport. So that's always my response. Like, you know, the world we live in is my sport. We're part of this community of chronic early adopters, which means we're probably always too early on a lot of things.

And that's fine. Which means like by listening to you, you're already early. Like this is, this is still early days and you can catch up on it later or you can get up, get up to speed on it. Now it doesn't actually matter, but still obviously have an opinion on where things will go in 10 years honestly i mean just from the

conversation the thing that thinks that stands out to me most is resilience and a creative mind in this future like if you have a creative mind and not just a skill set maybe that's something you develop and then resilience the ability just to see keep pushing beyond and persevere i think those are two skills that anybody can have, but specifically in this area, that's what comes to mind from our conversation today because you don't have to have the skill, you just have the creativity and the ability

to wield these future techs to do all the magic. But Swix, thank you so much. Appreciate you. We'll link up everything we can. Swix will get our gaps on the links. Hey, by the way, check the show notes if you're listening to this. There's probably going to be tons in there. I say probably because we haven't written them yet.

My hope is that they will be complete. But we also open source our show notes. So if there's something missing, if your name isn't Sean, Swix Wang, you're just a listener, you can contribute too. And we actually love that so much. So head to the show notes if it's not there and it needs to be linked up, fork it on GitHub, contribute it back. It's an easy win for you in open source. But Sean, thank you so much.

We appreciate you. Thanks for having me. Okay, thank you for tuning in. That's it. This show's done. That is the almost last episode of The Change Law this year. Thank you, Swix, for coming on

and taking us down this ai trip this ai road this ai future it has come by storm from chat gpt to stable diffusion to dolly to open whisper to all the things coming in 2023 wow if you have any questions for swigs drop them in the comments the link is in the show notes. Once again, a massive thank you to Fastly and Fly. And of course, thank you, Breakmaster Cylinder. Our beats this year have been banging. I love them.

Next year, even better. Looking forward to it. Here we go. And that's it. This show's done. Thank you for tuning in. New listeners, welcome. Old listeners, welcome.

And everyone in listeners, welcome. Old listeners, welcome. And everyone in between, welcome. And hey, ChangeLaw++ is our membership where we give you the chance to get a little closer to the middle. You make the ads disappear, you get discounts in our merch store, and you get access to bonus content on our podcasts. Check it out at changelaw.com slash plus plus. Okay, this year's done. I'll see you in 2023. Game on.