The Breakdown - What Is GPT-3 and Should We Be Terrified?

Episode Date: July 21, 2020

Today on the Brief: Mastercard, Standard Chartered and PayPal all deepen their engagement with crypto Japan inches closer to a central bank digital currency The real estate “doom trade” opens up Our main discussion: GPT-3 Generative pertained transformer-3 – or GPT-3 as it’s better known – absolutely took over the internet this weekend.  It’s a new AI language model that can do some truly incredible things, from writing poetry to composing business memos to generating functioning code from natural language descriptions. In this episode of the Breakdown, NLW provides a 101-level overview of GPT-3, including: What an AI language model is Why AI for language is more difficult than image-based AI The background of OpenAI, the Elon Musk-backed project behind GPT-3 Some examples of what GPT-3 can do Why reasoning and narrative still elude the technology Reference posts: GPT-3 Examples, a Twitter Thread Jonathan Johnson on AI Language Models Rob Teows: GPT-3 Is Amazing – And Overhyped

Language models pose a threat to disinformation. Where deepfakes are putting out fake videos of people, language models can be used by bad parties to create junk news or increase catfishing scams through fake text messages, emails, and social media status updates. Articles generated by a machine continue to be released that look like other people actually wrote them. What this means, effectively, is that in the same way that visual AI creates the possibility for deepfake videos and photos, and forces us to be even more on our guard about that sort of visual disinformation, this new language model technology should make us more on our guard about what we read.

Because the more that this AI gets trained, the more likely it is that some of the things around us are designed specifically to manipulate us and not just by good propagandists, but by actual machine propagandists. Welcome back to the breakdown, an everyday analysis breaking down the most important stories in Bitcoin, crypto, and beyond. This episode is sponsored by BitStamp and Crypto.com. The breakdown is produced and distributed by CoinDesk. And now, here's your host, NLW. What's going on, guys?

It is Monday, July 20th, and if your feeds were anything like mine over the weekend, you saw a huge number of tweets about GPT3, a new AI language prediction model that is just blowing people's minds. Before that, however, two quick things. First, if you like the breakdown, please rate and subscribe and review. Every rating, every review helps a huge amount, so thank you for that. And second, of course, let's do the brief. First upon the brief today, a crypto update with big players. So A huge number of big institutional actors upping their commitment to this space. First, MasterCard is expanding the crypto and fintech companies in its card issuance network, saying, quote, the cryptocurrency market continues to mature,

and MasterCard is driving it forward, creating safe and secure experiences for consumers and businesses in today's digital economy. A second example of this is standard chartered bank. They're building a crypto custody solution and say that they have at least 20 institutional investors who are already interested. Finally, following up on news from last week, that was really also news from last month, PayPal is reportedly chosen to work with Paxos crypto brokerage to launch their crypto service, which was reported, like I said, in June, but confirmed last week when we discovered a letter that they had sent to the European Commission in March saying that they

were working on crypto. Why did these stories matter? Well, it shows that in the future, consumer access seems unlikely to be the barrier, at least a main barrier to crypto adoption. If it doesn't get adopted, it's because people aren't interested and don't find it useful versus it's hard to use because you now have companies like MasterCard, PayPal, Standard Chartered Bank, all building in the space. I think that's obviously a bullish sign, but it does mean that the next set of barriers to push through have to do with use and value addition to people's lives. Second on the brief, Japan is getting serious about a central bank digital currency. So what happened?

A few days ago, reports came out that Japan's government was seriously considering including a CBDC in its policy framework, which would make it an official government policy. Just yesterday, Reuters reported that they have set up a new dedicated team that will expand the research that they've already been doing since January in collaboration with the Bank of England, the European Central Bank and others. So taken together pretty clear indications that Japan is upping this game. Why does it matter? Well, there's a couple different parts. First has to do with the global power dynamic with China. As I've mentioned before on this show, Japan is very nervous about China's influence in the region and their growing influence in the region, let's say.

They've said that the U.S. really needs to help them curb the influence of China, and they're very nervous about the central bank digital currency that China is currently testing as a tool for expanding that influence. So I think there's that regional dimension that has sort of a global geopolitical flair to it as well. The second reason why this is important, though, has to do with just our ongoing tracking of the Overton window on CBDCs in general. This is one of those things that I think is going to be research and exploration until all of a sudden the dominoes tip and every central bank in the world is racing to release their central bank digital currency. Last up on the brief is a narrative watch, which is something that I'm watching that is sort of

emerging and I think is interesting. So in this case, we're talking about the real estate doom trade. An article in Bloomberg came out today about how rich people are buying access to places like the Caribbean and New Zealand, effectively getting not just property, but also citizenship and rights in those places. This is something that George Gammon from the Rebel Capitalist show, who's been on this show, tweeted out about as well. He's in St. Barth right now and talked about how he can see how in a sort of increasingly dystopian urban environment in America or wherever you're from, wanting to go to these places was a very literal safe haven. I think this is interesting and relevant to watch, not just because it's sort of some weird wealth porn or

something like that. I think real estate is one of the wackiest, weirdest realigning markets that there is right now. Mobility is absolutely at a premium. The ability to move from one place to another has never been more valuable than right now. And ironically, it's also never been more constrained right now. An American passport has never given you less access to the world than it does right now. I think when it comes to real estate and just global mobility in general, or in a punctuated equilibrium moment that's going to be really, really powerful and important to how people organize themselves and societies around some of the most fundamental questions they face, which is where to live. If this is something that you're

interested in exploring more, real estate, whether it's in this global dimension, or even just the balance and shift in balance between urban and suburban and rural environments, let me know on Twitter at NLW so I can make sure to see that there's interest. It's something that I think would be fun to explore from the context of both society, culture, and macroeconomics, but if you're not interested, I obviously won't go that deep in it. So let me know if that is something that you would like to hear more about. And with that, let's shift to our main conversation about GPT3. First of all, this is a general caveat when I give you guys these sort of breakdown overviews of a new

topic or domain. I am not in any way pretending to be an expert. I'm just trying to synthesize what I've learned for you to hopefully give you the 101, the EL5, the TLDR, the whatever, right? The basics you need to engage with the content that you're seeing in a little bit better of a way. So GPT3 stands for Generative pertain Transformer 3. and what it is is a new AI language model.

Well, that might for you bring up the question, what is an AI language model? Jonathan Johnson wrote a great blog post on this on the BMC blog in April. He wrote that it's basically a statistical tool that predicts words and attempts to find patterns in human language. Up until now, AI has been focused primarily on images. Images are more accessible for modeling tasks

because they're, one, easier to label, as in we can say what's in a picture in a much easier way, and two, their data is already interpretable by computers because it's a set of pixels, right? It's much easier for a computer to understand a set of pixels than language, which has to be broken down into numbers in the first place. Images can be labeled for general tasks, while language models have to be developed for specific tasks. That brings us back to GPT3. GPD 3 is the most powerful language model ever built. GPT2 was released last year by the same company OpenAI

and was considered shockingly massive, having 1.5 billion parameters in the model. GPT3 has two orders of magnitude higher than that, 175 billion parameters. The company behind it, like I said, is OpenAI, and this was a project that was announced in October 2015 by Elon Musk, Sam Altman, who's the head of Y Combinator, and others who collectively pledged over $1 billion to the initiative. Now, Elon resigned his board seat in 2018 due to potential conflicts of interest, but continued to commit to funding the initiative. It has both a for-profit and a parent-n-profit, and the for-profit received a $1 billion investment from Microsoft in 2019.

Now, the company is reportedly very mission-driven, with their charter kicking off and saying, open AI's mission is to ensure that artificial general intelligence, AGI, by which we mean highly autonomous systems that outperform humans at most economically valuable work, benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but we'll also consider our mission fulfilled if our work aids others to achieve this outcome. As you can tell, AI is almost an inherently complicated, complex, and controversial area. It is inherently something that could disrupt the way that people work.

So any time there's any new AI, one of the first things people ask is, does it do X, Y, Z better than humans? And more importantly, perhaps, would a human still do better than it? Because in the answer to that second question, will be found what humans should be orienting themselves for in the future in some ways to be economically productive. Bit Stamp is the original global cryptocurrency exchange. Since 2011, BitStamp has been the preferred exchange for serious traders and investors,

trusted by over 4 million customers, including top financial institutions. BitStamp is built on professional-grade trading technology. Their platform is powered by a NASDAQ matching engine, and their APIs are recognized as the best in the industry. Download the BitStamp app from the App Store or Google Play, or visit bitstamp.net slash pro to learn more and start trading today. That's bitstamp.net slash pro. What's going on, guys? I'm excited to share that one of this month's breakdown sponsors is Crypto.com. Crypto.com offers one of the most cost-efficient ways to purchase crypto out there,

as they've just waived the 3.5% credit card fee for all crypto purchases. What's more? With Crypto.com's MCO Visa card, you can get up to 10% back on things like food and grocery shopping. When you buy gift cards with the Crypto.com app, you can get up to 20% back. Download the Crypto.com app today and enjoy these offers until the end of September. Let's get into the recent news and why this thing is all over everyone's feeds and really all over the internet. In May, OpenAI published their research on GPT3 and last week they started letting people actually access and play around with the model. The GPT3 API is in fact OpenAI's first commercial product and the text generated by GPT3 has been circulating everywhere on social media. This thing can write creative fiction, it can generate color,

It can compose business memos and ideas. So by way of example, I'm going to go through just a few of the things that have been done with GPT3 in just a few days. This is from Kai Sotala's thread on this topic. He writes about automatic code generation from natural language descriptions. Give me a page with a table showing the GDP of different nations and a red button, and all of a sudden that is built into code. Someone asked GPT3 a medical multiple choice question, and it responded this way. From a brief description, GPT3 correctly generates an explanation, indicating that it's a case of asthma,

mentions a drug that's used to treat asthma, the type of receptor that drug works on, and which multiple choice quiz question this indicates. This is on the creative side. Given a prompt with a few lines of dialogue, GPT3 continues the story incorporating details, such as having a character make 1800s references after it was briefly mentioned that she's a 19th century noblewoman. And then there's this crazy one

which you really just have to go look at for yourself, but it's called Dragon Model, and basically it's an AI dungeon game played with the new GPT3-based, quote, dragon model that involves a cohesive story generated in response to actions. Quote, the game invents a complex magic system and underlying theory behind why it works and describes the whole system as I read the book.

I'm going to read an excerpt of an essay from Manuel ARAS who wrote OpenAI's GPT3 may be the biggest thing since Bitcoin. Summary, I share my early experiments with OpenAI's new language prediction model. I explain why I think GPT3 has disruptive potential comparable to that of blockchain technology. OpenAI, a non-profit artificial intelligence research company, released its third generation of language prediction model into the open source. wild. Language models allow computers to produce random-ish sentences of approximately the same length in grammatical structure of those of a given body of text. In my early experiments with GPT3,

I found that GPT3's predicted sentences, when published on the BitcoinTalk.org forum, attracted lots of positive attention from posters there, including suggestions that the system must have been intelligent and or sarcastic, and that it had found subtle patterns in their posts. I imagine that similar results can be obtained by republishing GPT3's outputs to other messages. message boards, blogs, and social media. I predict that, unlike its two predecessors, PTB and OpenAI GPT2, OpenAI GPT3 will eventually be used to pretend the author of a text is a person of interest, with unpredictable and amusing effects on various communities. I further predict that this will spark a creative gold rush around talented amateurs to train similar models and adapt them to a

variety of purposes, including mock news, research journalism, advertising, politics, and propaganda. I was recently watching a podcast about how OpenAI built their latest language model, and it made me wonder what could be done with a system like this. I cannot stop thinking about the applications of such a technology and how it could improve our lives. I was thinking of how cool it would be to build a Twitter-like service where the only posts are GPT3 outputs. This system is an early prototype, and its behavior is not comparable of that of a real trained

AI. While OpenAI GPT3 does seem to be able to predict replies, it does not always predict replies to its own posts. nor do its predicted replies tend to be relevant or even grammatically correct. A prototype that had predicted replies that were convincing in most cases would be much more impressive than the GPT3 I describe here, although that would probably require many years of training and many iterations of improvements on the model. I am merely imagining what an open AI GPT3-like system might be able to achieve in the hands

of a talented human operator. So Manuel wrote this. There's a whole section as well about BitcoinTalk.org and how GPT3 interacted with it. And then you got this part. I have a confession. I did not write the above article. In other words, the article that I was just reading to you. I did not perform any such experiments posting on Bitcoin Talk.

In fact, I haven't used that forum in years. But I did it on my own blog. This article was fully written by GPT3. So basically, that whole article, that's longer than the excerpt that we just read, was written, in fact, by GPT3. GPT3 wrote an article about itself, and there were numerous other versions of this that were out there.

I'll link to a few of them in the show notes. So basically, how to think about it is this. This is a text predictor. It takes a chunk of text as input, then generates predictions on what the next text should be. So Manuel was writing this piece. He had it read his own articles, so it knew stuff about how he wrote,

and then it was able to produce this piece that sounded a lot like him. The thing is, it's really, really, really smart because it has ingested effectively all of the information available on the internet. It has half a trillion words that it's read. And so the question becomes, what can't it do? What are its actual limitations? There is so much hype and buzz. How should we actually interpret this right now? Rob Toos is a VC at Highland Capital Partners, and he wrote an essay about why it was both amazing but actually still overhyped. And this is what he had to say. He discussed two issues with it, the first was contextual understanding. He wrote,

GPD3 possesses no internal representation of what these words actually mean. It has no semantically grounded model of the world or the topics of which it discourses. It cannot be said to understand its inputs and outputs in any meaningful way. Why does this matter? Because it means that GPT3 lacks the ability to reason abstractly. It lacks true common sense. When faced with concepts, content, or even phrasing that the internet's corpus of existing text has not prepared it for, it is at a loss. A second piece, which is particularly relevant for folks like me, is that it can't do narrative. He also writes, a related shortcoming stems from the fact that GPT3 generates its output word by word based on the immediately surrounding text.

The consequence is that it can struggle to maintain a coherent narrative or deliver a meaningful message over more than a few paragraphs. Unlike humans who have a persistent mental model, a point of view that endures from moment to moment from day to day, GPD3 is amnesiac, often wandering off confusingly after a few sentences. The GPT3 folks said this themselves. They wrote, GPT3 samples can lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs.

Basically, it cannot reason, it does not understand the language it generates, and it can't do narrative. As I mentioned before, any time there is an advance in AI, you have to ask what are the concerns What are the questions that it brings up from a potentially negative impacts in society? And it's interesting because when OpenAI released GPT2, they said,

due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for research to experiment with, as well as a technical paper. Jonathan Johnson, the author that I quoted before from April, wrote, language models pose a threat to disinformation. Where deepfakes are putting out fake videos of people, language models can be used by bad parties to create junk news or increase catfishing scams

through fake text messages, emails, and social media status updates. Articles generated by a machine continue to be released that look like other people actually wrote them. What this means, effectively, is that in the same way that visual AI creates the possibility for deep fake videos and photos and forces us to be even more on our guard about that sort of visual disinformation, this new language model technology should make us more on our guard about what we read. Because the more that this AI gets trained, the more likely it is that some of the things around us are designed specifically to manipulate us and not just by good propagandists, but by actual machine propagandists. These cute experiments now where people are showing you how

GPT3 is writing about itself in the third person as though it were someone else are fun and kind of mind-blowing, but there's obviously much more significant political, social, cultural implications that are really serious. That's something that is a huge domain of human thought right now, that is the study of entire courses. I mean, obviously ethics around AI and what the world looks like in AI is one of the biggest meta-questions that we face. So certainly not something that I can wrap up in a nice little bow on this show. But hopefully you now have a slightly better sense of what GPT3 actually is and you can go engage with it as such. You can go learn all about it. There's no doubt that it's amazing technology and something that's really important to understand is now existing

in the world than people are going to be building with. There, I believe, are going to be amazing creative applications. They're going to be incredible business applications. And as with any new technology, it will force us to answer much more elemental and fundamental questions that we have about society. What is the obligation that people have to work in society when so much can be done with machines? These are questions that even without GPT3, we're going to be incredibly important for us to figure out. But now that this type of technology continues to exist and grow, they just get even more acute. Anyways, guys, I hope the show is helpful. I appreciate you listening. So until tomorrow, be safe and take care of each other.

Peace.