The Changelog: Software Development, Open Source - Twitter's open algorithm, Auto-GPT, LLMs as "calculators for words", SudoLang & stochastic parrots (News)

Episode Date: April 3, 2023

Twitter publishes (some of) its recommendation algorithm, Toran Bruce Richards puts GPT-4 on autopilot, Simon Willison shares a good way for us to think about LLMs, Eric Elliot creates a powerful pseudocode programming language for LLMs & I define and demystify the term "stochastic parrot".

What up nerds, I'm Jared and this is Changelog News for the week of Monday, April 3rd, 2023. Let's do it. On Friday, Twitter Engineering wrote about and open sourced their recommendation algorithm. People who previously decided Elon Musk is very good thought this release was very good. People who previously decided Elon Musk is very bad thought this release was very bad. Me? I thought it was just okay. I was in a band for a while. I was in a band for seven years. It was a death metal band. People either loved us or they hated us. They thought we were okay. On one hand, it's not the full algorithm.

In fact, the architectural diagram released with the code shows this big neural network in the middle called HeavyRanker, and the training data for that model is not part of the release. On the other hand, there is,000 lines of Java, 22,000 lines of Python, and 7,000 lines of Rust in the main repo alone. It's going to take folks a while to analyze all that's going on here, but a lot of weird stuff has already been found. One example is some metric tracking code that groups Twitter users into four types. Democrat, Republican, Power User, and Elon Musk. That is some weird, wild stuff. There's also some useful stuff that's been learned already,

such as how it weights tweet characteristics and engagement. By the way, likes are a 30x boost, retweets a 20x boost, images and videos are a 2x boost, according to Rowan Chung, who spent the weekend feeding the code into GPD4 because, of course he did. Perhaps the best thing this open sourcing produced, in my humble opinion, a treasure trove of absolutely ridiculous issues posted on the project's repo with titles like, There is Bug, Twitter is Pretty Cool, but Weird, My Cat Gave Birth to Four Kittens, and Join the Fediverse. Speaking of the Fediverse, I'm happy to announce that I'm open sourcing an approximation of Mastodon's recommendation algorithm right here on ChangeLog News. Are you

ready? Write this down. Select star from toots. Order by published at descending. Semicolon. That was far too much non-AI related content. I am so sorry. Here comes some large language models. Like, really large ones. Big fat guy. I mean like orca fat. Oh my, I just had an idea.

Maybe we can repurpose all of those Yo Mama So Fat jokes for large language models, and they'll still be funny without hurting anybody's mama's feelings. Hey ChatGPT, write me three funny Yo LLM So Fat jokes. Pronto. Yo LLM So Fat. It needs a search engine just to find its own information. Not bad. Yo LLM So Fat.

It can't even fit in the cloud. It needs its own weather system. Not good Yo, LLM so fat, it can't even fit in the cloud. It needs its own weather system. Not good. Yo, LLM so fat, it's the reason for global warming. Well, that's just depressing. AutoGPT is the new kid on the block. It's an autonomous GPT-4 experiment. You provide a name, role, and up to five goals, and AutoGPT will do actions to fulfill the goals all on its own. Not only is this a cool experiment in and of itself, it also has a breakthrough feature. It's able to write its own Python code and execute it. That means it can recursively debug, develop, and self-improve. Speaking of LLMs, Simon Wilson will be back on the podcast this Wednesday.

Adam and I chatted with Simon for a full two hours, and honestly, I think this episode's even better than the stable diffusion discussion we had back in September. After our chat, Simon wrote up his thoughts on how we should be thinking about LLMs. He calls them calculators for words, saying, quote, this is reflected in their name. A language model implies that they are tools for working with language. That's what they've been trained to do, and it's language manipulation where they truly excel, end quote. This makes large language models very good at a specific set of things, including summarization, question answering, fact extraction, rewrites, and suggesting titles.

Eric Elliott recently wrote about and released Pseudolang, a powerful pseudocode programming language for large language models. He didn't write it himself though, because that would be so passe. With GPT-4's help, Eric invented Sudolang, which has commands, modifiers, and arguments. The spec for the language is available on GitHub. All you have to do is paste it in the chat GPT with GPT-4 plus running, and you'll be off to the races. About the project, Eric says, quote, AI turns everyone into a conjurer, and we have only witnessed the tip of the iceberg. Have you heard people use the term stochastic parrot? I heard it a few times on our Practical AI podcast and didn't really know what it meant, so I looked it up so you don't have to. When people use this term in reference to a language model, they're referring

to the limitation of the model's understanding of the words it's spitting out. Stochastic refers to the random element in generating responses, and Parrot highlights the notion that these models merely repeat what they have seen during training like parrots do when you teach them to speak words. Here's Daniel saying just as much on practical AI. The model has no clue what it's saying and nor does it have any sort of like morality or like it has no clue what it's saying, right? It's just producing producing coherent output the same whether it's a text image model or it's a language model like there's no basis in like knowing what it's saying it's auto-completing text right like at the end of the day yes like that's a vast oversimplification and

you can not like that but but ultimately you're giving a prompt and it is producing output that is seeded by that prompt. Right. And it's just trying to produce this coherent output that's consistent with both the data that it's seen and maybe some type of extra mechanism like the human feedback or something that it's seen. But ultimately, it's just producing output that seems coherent and probable. It has no clue what it's seen. But ultimately, it's just producing output that seems coherent and probable. It has no clue what it's saying. The term comes from a 2021 paper called On the Dangers of Stochastic Parrots. Can Language Models Be Too Big? Dr. Joe Ancora summarizes it as a stochastic parrot has form, but no underlying model. We ourselves create the illusion of an underlying mind. That is the news for now. We'll be back in your ear holes on Wednesday when we publish our conversation with Simon.

Have a great week. Share Changelog News with your friends if you dig it. And we'll talk to you again real soon.