The AI Daily Brief: Artificial Intelligence News and Analysis - Can Open Source AI Compete with Big Tech? A Look at TruthGPT, RedPajama and MiniGPT

Episode Date: April 18, 2023

Some argue that open source AI is the key to making AI's benefits to the entire world, as well as making AI safer. Others think that open source can multiply risks. With a slate of new projects being announced this week, the conversation is heating up. Discussed in this episode: Elon Musk's planned TruthGPT  Dolly 2.0, an open source LLM based on the EleutherAI pythia model built by Databricks RedPajama, an open source proxy of Facebook/Meta's LLaMA, from Together  MiniGPT, an open source AI model that can extract information from images  Stable Diffusion XL, the latest open source text-to-image model from Stability AI

The episode of the AI breakdown that you're about to hear first premiered as a video on YouTube on Tuesday, April 18th. The discussion is all about Truth GPT, Red Pajama, Mini GPT, and the slate of other open source AI tools and models that are being released now and making people wonder whether open source AI can compete with big tech. Welcome back to the AI breakdown. Today we are talking about open source AI. whether it can compete in a world of intensely lucrative commercial applications and all of the capital infusions that comes with them and the cost of data, et cetera, et cetera. And there are a lot of reasons why this is an important conversation. There are many debates about whether AI being open source would solve some of the problems

that people are worried about in terms of commercial concentration, in terms of even some of the safety risks. But at the same time, there are others who think that open source AI might introduce new risks because it's a less controllable type of experimentation. And I want to get into this because it's a conversation that's increasing right now for a few different reasons. There is first Elon Musk and his announcement of TruthGPT or what he wants to build called TruthGPT. There are a couple of LLMs that have been announced in the last week or two that are trained on large, large data but that are intentionally and very mission-driven open source, including Dolly and Red Pajama. There's some really interesting little applications that are actually outperforming other closed

competitors, notably mini GPT, which is a chatbot that can actually have a conversation about an image and et cetera, et cetera. There are lots of projects right now that are making this conversation or putting this conversation right at the center of the discourse. So with that, let's talk first about Elon Musk and TruthGPT. So where this really got accelerated, TruthGPT is something that Elon has mentioned in the past. He has obviously been around the AI conversation for some time. He registered X.aI. It seems like a space that he's inevitably going to get into. But he was on Tucker Carlson this week, and he specifically was talking about AI in a much more intentional and kind of pointed way in a way that suggests that

he was going to do something. So he's discussing in this interview some of his concerns around Open AI, which he was a donor to, moving from an open source nonprofit approach to a closed source for profit approach. He talks a little bit about his concerns around the kind of wokeness training model of Open AI and why there needs to be something that isn't trained in that way. And he also thinks that there are just different approaches that might lead to better outcomes from a safety perspective. So Nathan Lanz here has a little thread about everything that Elon said. And basically, he summarizes that Elon suggests that he wants to design a, quote, maximum truth-seeking AI, which would be a vision of an AI that serves humanity.

Elon believes that by getting his version of this to understand the nature of the universe, it will become less likely to harm humans, as we are part of the universe that interests it. Now, this could be a very naive point of view. That's certainly what some have pointed out or suggested. It also could be something where there's an inherent politics to it, right? That many liberal commenters in particular are concerned that he's just going to, quote, feed it a bunch of right-wing BS. and call it truth. Obviously, that comes or is colored by the way that they've interacted with Elon in the context of him taking over Twitter. But what happens when Elon gets involved in something is that everyone pays attention. Mainstream media starts picking it up. And so I think at the very least, and the thing that you can be optimistic about, even if you're not particularly keen on

Elon, is the idea that there will be a bigger place for this conversation, given that he is getting involved. So that's one part of why this open source conversation is coming up, because Elon and TruthGPT are wading into it. Next, there are these LLMs that are coming online that are open source and alternatives to closed source models. So the first that's notable came out or was announced last week. It's Dolly or Dolly 2.0 really. It comes from a company called Data Blocks, and they call Dolly the first open source instruction following large language model for commercial use that has been fine-tuned on a human-generated data set and is open source. Dolly 2.0 is a 12 billion parameter model. But then there's Red Pajama. This is the

one that really has people talking this week. So this is from Together's website. Together is is the company that's unleashing this project. They say Red Pajama, a project to create leading open-source models, starts by reproducing Lama training dataset of over 1.2 trillion tokens. Now, Lama is, of course, Facebook's large language model or their family of models that was announced a month or two ago. I've lost all track of time. And for anyone who's apparent, you will get the joke of calling this red pajama as opposed to Facebook's llama. But what is red pajama? Well, we've got a couple summaries that help us understand it.

Ben's Bites, who writes a daily newsletter about AI, writes, Red Pajama is the biggest open source rival to big tech yet. They go on. You've got to love the determination of open source to go toe to toe with big tech. It reminds us of all that game-stop stuff a few years back. And Red Pajama is the latest initiative looking to drive this forward with bold goals. They have a three-step plan to rival the big dogs with fully open-source, reproducible leading language models.

That three-part model is one, produce a dataset, two, train a suite of base models, three, implement instruction tuning. They announced today the completion of stage one, a 1.2 trillion-word dataset, mimicking that used to train Lama. With promises of fully trained base models released in a matter of weeks and apparent progress on instruction tuning with Open Chat Kit,

we can feel the trembling of something big in the pipeline. Now, Pete, non-Mayer Pete, who is the writer behind the neuron, went even farther. He says, together is announcing Red Pajama, a fully open source project to replicate Lama. We will soon have a chat GPT that is completely owned by the people

versus controlled by OpenAI. This launch will go over most people's heads, so let's do a quick explainer. Now, Pete goes on to explain that meta released Lama, a family of models that can beat GPT3 in March, and the Lama blog post, he says, talked extensively about the importance of giving more people access to models to make them safer. They open sign-ups to researchers and gave access pretty widely. But Pete writes, Lama is released under a non-commercial license, which means you can't use it beyond toy purposes. Now, what together and Red Pajama are then doing is trying to make this model or make this type of model. Remember, it's a 1.2 trillion token data set that replicates Lama, completely open source and open for all research and commercial use.

Pete says these will unlock another explosion in open source chatbots. Now, this is a big deal because of, one, it opening wider access and two, because when you have this, sort of open source, it allows more people to build more things faster. So Red Pajama is a huge driver of the conversation this week, even if it's not getting perhaps as much attention as something like Auto GPT was last week. But I think there's another story or another project that is also driving people to get interested in this. And that's Mini GPD. So Mini GPD is an open source AI model that allows you to upload a photo and start chatting with GPT4 about the photo. So for example, the little demo that they have in one of these tweets about it is things like

showing a picture of a lobster roll and generating a recipe from it, showing a whiteboard write-up of a website description and having it start to generate the code. Having a rhyme generation prompt where there's a picture of a woman and her. dog and the person interacting with mini GBT asks for them to write a poem about it. So that's the type of thing that this can do. And importantly, this is a technology that has long been promised as a part of open AI. It's something that open AI has said they want to do, but they haven't released yet. So the fact that we're getting an open source version of it before the open AI version of it even comes out is to a lot of people incredibly encouraging. It is, by the way,

also just a phenomenally interesting technology. But the other one project that I wanted to mention is, of course, stable diffusion. Stable diffusion announced Stable Diffusion Excel beta available over the last week. Stable diffusion is intently focused on open source development. And so given how much excitement there is around their new Excel model, that is obviously driving this conversation as well. Now, there are a couple reasons why this is something interesting to people. And I think one is just the builder mindset and the commercial reality of it. So there's a scale state of AI report. And one of the questions they asked was, how do you work with generative models? And 28% said open source only. It was the most popular answer. So there's clearly a lot of

demand from developers and from organizations to have these sort of open source models. But then there's also the larger questions of what open source models do for the larger world of AI. Sacrificial pancakes writes keep embracing and encouraging open source AI. Inoculating the public against the new advances and the rapid pace of innovation is the only way for humans to have any say in what is to come. If we play our cards right, an entirely new world is possible. Now this is a take that I've seen quite a bit on Twitter is that open source actually deals with or addresses some of the questions of the rapid advances in AI because it creates a level playing field. It makes sure, in other words, that the innovations of AI are not distributed only to the people

who are powerful enough to have access to the small number of closed source tools. Now, the question, of course, is whether that actually deals with all of the issues of AI, right? You kind of have really big and different conversations around the challenges of AI. On the one hand, there is the issue of access and who these new tools benefit, which is sort of where people think that open AI can be a, or open source AI, let me be clear, can be a real benefit. But then there are also other folks who are worried that they make the safety issues, the X risk issues, much more profound. So for example, Ethan Mollick writes, I've been playing with the various open source efforts to give AI access to other systems. On the one hand, they can make

AI much more useful and powerful. On the other, this is something the open AI GPT for white paper cautions about, as it multiplies the potential for unexpected risks. Ethan is a professor at Wharton who studies innovation and startups. Now, the thing that he's talking about is, for example, all of these new open source auto-GPT style efforts that allow AI to interact with the internet, with other AI platforms and data sets, those things are potentially each a little Pandora's box that creates new challenges. But again, we're left with this question of do we only allow a small handful of certified experts to be the ones to take those risks or do we level the playing field of risk? And I think it's a really important question to ask. There's another person

who tweeted, I think the government should require AI companies to open source their parameters. And it doesn't seem to me impossible that that's a place that we might go. So let me try to wrap up here. I think over the course of the next few years, certainly, but probably even faster, because I don't think it's going to feel like something we can wait on. You're going to see a massively growing conversation about AI in the public sphere. You're going to see a regulatory conversation. We're already having the Biden administration floating test balloons.

You're already seeing China come with their sort of regulations. it's going to happen. And so one part of the conversation, I believe, will likely be around to what extent open source and just being open with the way that models are trained and the parameters used within can make a difference with these issues. Anyways, guys, that is the reason that we're talking about open source AI right now. It's never a conversation that's too far from anyone's mind. But pretty notable that you have this big movement of all these projects.

like Dolly and Red Pajama, that are really making it be a different sort of competitor to close source AI. That's it for the AI breakdown today. Appreciate you listening. Peace.