The Vergecast - The chatbot becomes the teacher

Episode Date: September 22, 2024

For the first episode in our new miniseries about the impact of AI in our everyday lives, we chat with Steven Johnson, a longtime author who has spent the last couple of years at Google working on an AI research and note-taking tool called NotebookLM. We talk about whether AI can really help us learn better, how Google has tried to make NotebookLM more accurate and helpful, and whether AI-generated podcasts are the future of learning.  Further reading: NotebookLM Steven Johnson’s website / newsletter From Steven Johnson: Listening To The Algorithm Google teases Project Tailwind — a prototype AI notebook that learns from your documents Google’s AI-powered note-taking app is the messy beginning of something great Google is using AI to make fake podcasts from your notes Email us at [e-mail] or call us at 866-VERGE11, we love hearing from you. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Welcome to the Vergecast, the flagship podcast of Infinite Context Windows. I'm your friend David Pierce, and this is the first episode in our new mini-series, all about AI and real life. We did a few episodes on this subject earlier this year, and it continues to be a thing that we're talking and thinking a lot about. For all the big, heady talk of how AI will either change everything or kill us all or make nobody ever have to work again or make us all have to work training robots, what is any of this actually good for, like right now?

That's what we've been trying to figure out. For today's episode, I'm talking with Stephen Johnson, who is a personal favorite author of mine. He's written 14 books over the years, and he actually told me that he can name them all in order off the top of his head, which I believe and also find very impressive. And some of those books are books you've probably heard of, like where good ideas come from and how we got to now.

But in addition to all of that, Stephen has also spent the last two years working at Google on a project called Notebook LM. Notebook LM, if you've never heard of it, is an experimental thing out of a team called Google Labs. It started out as a thing called Project Tailwind a couple of years ago, and the idea has always been to make an AI-powered tool basically for making sense of your notes. In Notebook, which they all call Notebook, so I'll just start calling it Notebook, you first upload a bunch of documents or links to websites or PDFs or whatever else. and the tool builds sort of a corpus of stuff. The idea is you put a bunch of related things into a notebook in notebook. Then you can ask questions about those documents,

or you can have notebook build you an automatic study guide or an FAQ of the information in those documents. You can have it find you stuff in those documents, all that kind of stuff. It's sort of a note-taking tool, but mostly it's a research tool. Stephen calls it a tool for understanding things, which I like a lot. I've been covering and using notebook for a long time, but I wanted to have Stephen on now because notebook is kind of going legit. I assumed, if I'm being completely honest with you, that it was like a neat experiment that

would eventually die because everything dies at Google, or at best, it would just be a tiny feature buried in a menu of Google Docs or something. But notebook is growing and it's expanding and it's actually starting to do some really interesting new stuff. Recently, they launched a new feature called Audubon. audio overviews, which generates a podcast hosted by two chatbots based on whatever documents you upload. It is wild. Actually, you should just hear this.

So I made a notebook in notebook with a bunch of stuff from the ongoing U.S. versus Google ad tech trial. And here's just a few seconds of the podcast it generated. Okay. And that's where header bidding enters the picture. Ah, header bidding. Yes. This was their attempt to kind of outmaneuver Google. Like finding a side entrance into the auction.

That's a great way to put it. So with header bidding, publishers could essentially offer their ad space to multiple ad exchanges at the same time. Look, I don't know if that's good or bad. I don't know if all of that information is even true, but I'm fascinated by the idea of a tool that tries to make it easy and automatic to learn almost anything in whatever style works for you. And Stephen is just as fascinated by it. So I figured we should talk about it. All that is coming up in just a second. This is the Vergecast.

We'll be right back. Support for the show comes from Retool. Too many companies run critical operations on duct taped spreadsheets, Slack workflows, and whatever else they could cobble together. Not because they want to, but because building internal tools means weeks of waiting on someone else's backlog. That's where Retool comes in. Build custom internal tools just by describing what you need.

Prompts something like, Build me a revenue dashboard on our Salesforce data. And Retool actually builds it. On your company's data and your cloud with enterprise security built in. Go to retool.com slash Verchcast. We all need to retool how we build software. All right, we're back. Let's get into my conversation with Stephen Johnson.

I figured the best place to start with the notebook LM story was just at the beginning. So that's where we started. How is it that an author comes to be a Google employee working on an AI product? Yeah, it's a really interesting story, I think. So while I have spent most of my career writing books about kind of science and technology and history, I've always had this kind of side interest in using the technology to help me with the book writing process, with the research process, you know, tools for thought, that whole tradition has been a big influence on me. And I've always been kind of an early adopter of, you know, I use this program called Devon Think,

organizing all my notes and quotations from books that I'd read in the 2000s. Well, that's a whole other Vergecast we're going to have to do at some point. So get ready for that. Yeah, yeah. So, you know, I wrote about it and kind of blog posts, and I wrote a couple things for the times about this. And it shows up in my book where good ideas come from. I talk about using tools like this and was a big Scrivener user and evangelist and

all that stuff. So I've always been interested in the software side of writing and thinking and research. And in the spring of 2022, you know, so six months before the chat GPT moment, I wrote a very long piece from the Times magazine about language models in general. It was effectively just making the argument, like forget about AGI or any superintelligence or anything like this. These models have basically learned how to communicate in coherent language and they understand what we're saying. And whatever else, like this is going to create all these new possibilities. Yeah. That piece was very good, and I remember a bunch of the response you got to that was from people being like, this is bonkers, he's out of his mind, there's no way any of this stuff is going to be that big. And then, boy, did you time that correct.

Well, I did accept that it was painful. Like, I'm very conflict-averse, David, if I can tell you. Be honest with you. And there was, I mean, a lot of people did like that piece, and I think we're inspired by it. But there was definitely a lot of comments from people saying, like, oh, he fell for the height. You know, this stuff is just auto-complete on steroids, and I can't believe he's so naive that he got excited about this thing. The piece addressed a lot of the objections and criticism and took them very seriously, but it was like, the one thing you can't do is dismiss this technology. Like, something fundamental has just happened. And we're going to spend years figuring out, like,

how we apply it and how we deal with the upside and the potential downsides and, like, just take it seriously, people. So, you know, if I timed it later, I wouldn't have maybe had as much of the pushback, which was hard to. I couldn't enjoy that piece going out of the world. Let me put it that way. Well, it was a good piece.

I liked it. Well, I appreciate that now. So right around that point, kind of a new division inside of Google called Google Labs, there had been a previous Google Labs, and this is kind of a new one, was getting spun up. And at the time, a guy named Clay Bevoir was running it, and Josh Woodward, who runs it now,

is talking back and forth with Clay. And Labs had this kind of ethos of the division was basically, we want to create a place where we can do faster and more nimble product-focused experiments with new technologies. It's not just kind of research, but it's not working within the existing mature products. And when something new comes along, we can experiment and build things very quickly. So somewhere in between the like 20% time project and the like full-blown new Google product. Yeah, it was just this little hole that didn't quite exist. And there were so many interesting new technologies coming out, particularly.

the language model is that it seemed like this was a time for a lot of experiments to bloom, right? And they had had this idea that maybe they could also have an ethos of co-creation where they bring in outsiders. So if they're making a product that involves music, there should be a musician in the room from the beginning.

And it's not something that they build and then they show to the musicians or they have, you know, UX interviews with musicians. There's actually someone there who represents that kind of profession in the room for the life of the product. And so I was kind of the guinea pig for this approach.

And so they reached out, Josh and Clay had read some of my books and I read that Times article. And so they reached out and said, hey, you know, you've been dreaming of this ideal software tool that helps you organize your thoughts and helps you write and helps you formulate connections and, you know, brainstorm. We think we can now do it with language models. Like this thing you've been chasing literally, you know, I've been chasing this since I was in. in college in the late 80s when HyperCard came out, you know, for the Mac. Like, I'm an old person. I've been after this for a long time. And they were like, look, I think if you came to Google, you know, come part-time initially,

and we have a small team and we can, you know, we can build something. And I thought that sounded like an amazing journey. I honestly, I thought we'll build a prototype. It'll be fun. I'll meet some interesting people, but nothing will come of it. But it was still, you know, a fun ride to go on because of my passion for this. And then we built something that was originally called Tailwind, Project Tailwind, but from the beginning,

from very, like day one of the idea, there was always this sense that this was not just going to be an open-ended conversation with the language model. It was always going to be about the model being grounded in the sources and the information that you gave it. And it was really about respecting the original kind of human authored information, whether it's a book or your own notes or a scholarly paper or your syllabus for your class, and basically saying to the model, take this information, which is personally

relevant to me and is verifiable in some way or factually, you know, trustworthy, and base your answers and everything I asked you to do on that information. And that was the seed of the, you know, We had a version of that in August of 2022, like, you know, on my fifth day at Google. It preceded me. I should say there was a program called Talk to a Small Corpus that was about a month underway when I got there. And then one of the first things we did, we put in my book, Wonderland. And we just, like, I had this experience of kind of chatting with the model and having it answer based on information in my book. And, you know, that was one of those moments you're like.

A lot of possibility just opened up. Well, actually, the weirdest moment I would say was a little bit later, over Christmas break, the internal version of what became Bard was kind of released to some of us inside. And over Christmas break, I was spending a lot of time with Bard. My family had gone off skiing somewhere at I don't ski. So I was just like home alone. And Bard came out. I was like, well, I just now have like 18 hours a day to talk to this.

My new friend. My new friend Bard. So I would occasionally just start with just a little preamble to get its bearings. And I would say, I would like to discuss Stephen Johnson's book, The Ghost Map. And this was just based on its kind of training data. This isn't with source grounding. And so one day I do this, and I was like, oh, yeah, I would love to discuss that. That's a fascinating book about medical mystery in the 19th century that explores the impact of cholera and epidemiology on the history of London and the history of cities generally.

And so I'm like, oh, well, thank you very much. I'm actually the author of that. It's me. And Bard said, oh my gosh, I am so sorry. I can't believe I didn't recognize you, Mr.

Johnson. Oh, wow. And I was like, I know, but there was no way you could have recognized me, right, Bart?

But it was just one of those moments where I was like sitting alone in my study, having this conversation with an algorithm that's apologizing for not recognizing me when it's a fan of my book.

And at one point it said, I'm just so excited, at the opportunity to get to work with people like yourself. And I was just like, this is strange. So there were a lot of uncanny moments like that. But in a way, you know, there was part of that that I also recognized was an illusion, right? It was trained on the way that people react when they meet people or fail to recognize someone. And so it responded in that way.

It obviously had no inner life. It was not actually embarrassed by the fact it was meeting me. It was just kind of play acting at that. So to me, the stuff that really was mind-blowing was just its ability, and this really kicked in, you know, for us when we switched to Gemini, its ability to extract and see patterns in large amounts of information. You know, I have this notebook where we have something like, you know, almost a million words of transcripts from the NASA Oral History Project.

So it's just interviews with like the NASA astronauts and flight directors and things like that. And you can go into it and say, I'm interested in, you know, I'm working on a documentary about the early Apollo program. And I'm interested in the emotional connections between the participants, particularly the astronauts. So can you create a detailed guide of all the points in these transcripts where anything that seems interpersonal or emotional comes up? Give me a summary of that section. giving me a direct quote from the section and of course include citations so I can click immediately

and go back and read the passage in its original form. And it will just do it. It'll take a little bit of time because it's a complicated query, but it can take something that would have taken 40 hours to compile that document.

Like how, I mean, you know what it's like working with information like this? It will generate an incredibly convincing and accurate first draft with grounded citations to all the passages in about 45 seconds maybe. And so it's literally, you know,

a thousand times faster than it would have been before to do that kind of thing. And it's not pretending to be a person. It's not pretending to have feelings about it. It's just grabbing that very subtle kind of collection of information that is not anchored in any keyword, right?

You know, it's not looking for mentions of emotion. Like, it just understands that these other things where they talk about their kids, that's an emotional moment. And it's able to kind of co- that way. When it started being capable of doing that, that was the point for me where I was like, oh, this is really a fundamental change. And that was, that happened with Gemini, which was what, earlier this year? Yeah, we switched to, we switched to Gemini 1.0 in December and it was great,

but it was really 1.5 pro and the bigger context window. I mean, the other thing that happened to me, by the way, I'm a nice writer, dude. And now, like, I hear like a new million token model and it's like the most exciting thing in my life. I cannot wait to get my hands on it. So when we got first access to that model, I took the entire text of my book, Infernal Machine, which just came out a couple months ago, but was in manuscript form at that point.

So this is really important. None of the words from my book were in the training data for the model itself. So it had never been published. It had never been discussed about in any coverage. So the facts, it's a book of history. So the facts are probably, in some form in the models training data,

but the book itself and the way that I presented the facts was not. And the thing that had struck me about the early discussions of large context was that people were using it to do these kind of needle in a haystack test where they're like, oh, we gave it, you know, the full text of Moby Dick, but we added this one line that was different and it was able to find it, right? And which is cool, and you know, you couldn't do that before. But the point that I was so obsessed with is that once you have the full text of something like a book in context. It means that the model can find obscure things in the

text, but it understands the sequence of the text. And it can understand large kind of movements of like cause and effect or change over time in a document, which you can't get if you're just giving it isolated paragraphs and snippets of things. And so I put infernal machine into this version of Gemini, and I basically asked it inside a notebook, L.M. I was like, give me, I was like, I'm interested in the way that Johnson uses suspense in this book. I would like you to list four places where he deliberately withholds information from the reader in order to peek their interest, describe the passages where he does this, quote from them, and then explain the future information later in the book that he's obliquely referring to that doesn't arrive for another, you know, 50 pages or whatever. And it just absolutely nailed it.

The first example it gave was exactly what I would have picked is the ultimate kind of form of suspense, which is an allusion to a ticking, mysterious ticking suitcase that happens in the preface. The mystery behind it doesn't get explained for another 200 pages. And so, like, think about that as like a search query. Like, search is find examples where something isn't mentioned and isn't mentioned in a very provocative way. And then fill in the blanks of, you know, the things. thing 200 pages later that it's obliquely referring to. Again, the model is not understanding the book on some level because understanding is

a word that we associated with consciousness and with sentience and with the inner life of what it means to read a book. But the model is doing the thing that human understanding does. Yeah. You know, and that's an important distinction, I think. No, I think that's right. One thing I heard somebody say not that long ago is that a meta, the better metaphor they liked than its understanding is just that it can see the whole thing at the same time.

Yeah. And I always thought that was really great. It's like if I can see one page at a time, this thing can see all 300. And it doesn't, it's not better at knowing those things. It's just literally by being able to see it all at once. Yeah. The number of things that suddenly become very basic because you can see them all together is very powerful.

And I just, that makes it both sort of simpler and cooler all at the same time, which I really like. But you bring up this tension. that I think is fascinating with all AI stuff, which is that there is a set of things that are just sort of remarkable that they're possible, right? And you see this with every new model that comes out and every new product that comes out. One of the first things everybody does

is just try wild stuff. And some of it works and some of it doesn't, some of it's amazing and some of it's dangerous and whatever. So there's the sort of novelty factor of it that I think is still so rampant in everything AI right now. And then there's the question of what is any of this actually for? And I think one of the things that I've liked about Notebook and sort of watching it develop over time is it feels like your sense of not just what this can do, but what it's for has gotten much better over time. And I wonder if part of that is like, does the novelty of moments like that start to wear off and you start to realize like, okay, that's cool.

But no academic is actually going in here searching for what's missing from this book. Like that's not like a thing most people need in their lives. And you start to sort of wind it back to like, okay, how do we bring? bring that sort of enabling technology to things people actually do need, or maybe is the craziness the point, and we've just never been able to do it before. So now we're trying to discover it all all at once. So I guess that, especially in those early days, really before Jam and I 1.5 kind of lights your brain on fire. Like what is that process of figuring out not just sort of what can this thing do, but like what are we building this tool for? Yeah, it's such a great

question. So many different ways to get into it. I mean, I think in the very early days, there was a sense that we were building something we knew was not going to really work because the context wasn't big enough, the model wasn't sophisticated enough, but you could see where things were going. And so much of what we really focused on from the beginning is like,

what is the proper interface for this kind of thing? Like if, you know, it's not just about a text message thread. Like surely there are other kinds of forms of UI. So if you were going to build a product from the ground up knowing that it was going to be built around a language model, like, what would it look like? Let's start from scratch. And that's a very fun, open canvas to have, but it also means you make a lot of stuff that it's not very good.

It doesn't really make sense. And it doesn't work because the model isn't caught up to it yet. So there was a lot of experimenting with that. I think we had an advantage in the early days in that I was just trying to drive it towards my very specific use case of like, I want a thing that has read all of my work and all of the quotes from books that have influenced me and can be a second brain to help me, like, remember those things and make connections. And so kind of author, researcher mode, and that we kind of built the first prototype

with that. And then once we were able to kind of open it up to more users, I think then we were just constantly discovering all these amazing things. And a big addition, this was Riza Martin, our product manager, who, you know, has just been, she's just an incredible, like, listener to users. She set up this Discord that, you know, was so central. Like at the very beginning we had a Discord, which is not a normal thing to do in some ways. And we now have this amazing community.

And it's constantly filled with people being like, oh, yeah, I saw this opportunity to use it in this way. So, like, our favorite one that completely came out of the Discord was Dungeons and Dragons players started using it. Because they were like, I have these big, you know, campaigns that I've designed that, you know, I'm a dungeon master. and I have, like, created this whole virtual fantasy world, and it's filled with all this information, and it's hard to keep track of. But I can load these documents into Notebook L.M. And then I can just, like, ask any kind of open-ended question,

and it'll be like, how many hit points do I need to kill this orc or whatever it is? I'm not a D&D player. And they were using it that way, and people were writing fantasy, also, like, world-building kind of fantasy novels, and they just need, they had a kind of story bible with all their characters and the backstory and everything like that. And it turned out that, you know, they'd never really had, an interface that let them do that. And that was not something we were thinking about it at all.

So we've just been like, you know, once we got past that first little prototype stage, we've just been really listening, like, intently to like where people are trying to push the tool and then just like making it easier, making it so they don't have to push quite as hard to use the tool that way. And that's, yeah, that's where we are. Well, it's funny. I mean, even thinking about you, you've written a couple of times over the years about your, you're sort of of endless, I think you call it the Spark File, that is basically just like a mountainous document of all the good and bad story ideas you have. I'm paraphrasing, but I think that's right, right? They're all good. I don't know what the bad ideas. You're talking about, David. That's

flawless, perfect by that book now. Many bad ideas. And I think, like, just listening to you describe the story Bible for the world building stuff. Like that, that actually is like a perfect down-the-middle use case for this in a way I hadn't even really thought about until just now, that it is like, here's a bunch of stuff that I have decided, one way or another, right? Like, here are my inputs. Help me make things with that. It's actually, like, kind of an amazing

and very difficult otherwise use case because it's like, oh, yeah, how many I have to go and collate that piece of information, that piece of information. I think there's all kinds of, like, complicated things with how we think about the art on top of all of that. But that thing where it's just like, I want to tell you the rules

and I want you to help me make games out of it, feels awesome. Like, I'm so much less conflicted about that than I am about so many things in AI. That's so cool. I love that. Yeah, well, we're trying to do the things that are less conflict. I said I was conflict-averse, so I've just like tried to steer towards this thing. But, yeah, it's a great point that kind of like take this mass of unstructured data and turn it into a set of kind of formats that help me do the job that I'm trying to do with some guidance for me. And this was, by the way, this was another great place where like Riza really saw this before I did, because I was thinking of it as, I'm going to write my book. So I'm going to do all the,

like, content creation here. I just need to be able to, like, surface the facts and make some connections, you know. But it turns out there's just all these places where you've got all your company documents and you want to create an FAQ for new employees. Like, no one is going to win a Nobel Prize for literature for creating that document. Like, and if you can get a first draft to that document in 45 seconds instead of in four hours, like, that's a win, right? That is good news. And so there are all these different workflows that are out there where there's massive information needs to be kind of like filtered in some way and turned into something else.

All right. We've got to take a break. And then we will be back with more from Stephen Johnson. We'll be right back. Support for the show comes from Framer. Framer is an enterprise grade, no code website builder. used by teams at companies like Perplexity and Miro to move faster. With real-time collaboration and a robust CMS,

with everything you need for great SEO, not to mention advanced analytics that include integrated AB testing, your designers and marketers are empowered to build and maximize your dot com from day one. So whether you want to launch a new site, test a few landing pages, or migrate your full.com, framer has programs for startups, scaleups, and large enterprises to make going from idea to live site as easy and fast as possible. Learn how you can get more out of your dot com from a Framer specialist or get started building for free today at Framer.com

slash verge for 30% off a Framer pro annual plan. That's Framer.com slash verge for 30% off. Framer.com slash verge. Rules and restrictions may apply. All right, we're back. One of the things I've been tracking about Notebook for a while is how it approaches things like hallucinations and citations. Because this isn't like a silly chatbot, you know, it's a research tool. It's not silly or acceptable or just a signal that it's early if it's wrong. It's a problem if it's wrong. It's useless, frankly, if it's wrong. So I asked Stephen how they're trying to solve that very real problem in AI. So before we get too far away from it, I do want to talk about the accuracy

and the citations bit of it, because you signed yourself up for a pretty high bar on both of those things, both by virtue of the people who are going to use your product and just what the product is, right? You don't really get to have the warning at the bottom of everything that says,

this thing makes some information up. Like, its entire purpose is not to make information up. How do you, what have you guys done? I know Notebook was an early experiment in in RAG, which is a way of basically winnowing down some of these systems. But like, what, what have you guys done differently at Notebook to try and solve that? And I'm curious, both on the underlying tech side and on the user experience side. Yeah. Well, we tried, I kept calling it

source grounding because I think that is a better name than RAG. I don't know, man. We all know GPT now. So I've just, I've given up. We're all doing the acronyms now. So, you know, I think Part of it was the fact that we were doing it from the beginning, that it started with that. It wasn't like, oh, let's build a chat model and, oh, shoot, we need to be able to like, grounded in other documents. It was from the beginning we were doing that.

And so we just had a lot of time to iterate and explore. The Gemini models are really good at source grounded. There's a lot of training sets. We contributed a bunch of them to just given this document, answer questions factually based on the information that documents. So we inherited like a great tool that, you know, we did very little, you know, kudos to the Gemini team for building a model that is much more faithful to the source material

you give it. But we built it also, you know, this is one of these things where it's like it's underlying model plus the UI. It has always been like our mantra for the beginning. And, you know, both things are required. And so one of the key things that we've had. pretty much from the beginning is that you can always read your sources in the app. And then with the release in June, we switched it over so that you now have inline citations to everything.

So anything the model says has a little link. You can read the original passage if you hover over it that shows you, you know, the source material for that. And you can click on it and you can go read the source in the app. Do you think that's enough? We've talked about, so one of the things you can do right now, actually, with notebook that we want to actually turn into a proper feature, but you can do it right now, is you can upload a bunch of, you know, kind of source material, factual source material, and then you can upload the article you're writing, for instance, and you can say, fact-check this article based on these sources and suggest

improvements. Oh, that's clever. That's a good idea. And, yeah, it's amazing. And it will go through and be like, well, this is correct. This is potentially wrong. It will suggest, and it'll have links to everything. And so to me, if I felt that the model were just hallucinating wildly in its responses,

then I do not believe that, you know, just providing citations and the ability to kind of fact-check manually and go back and see the original passage would be enough. But we've, you know, we've been sitting there banging away like quality reviews, like, constantly for the last year and a half. Like we have a poll, you know, sheets and sheets of sample questions. and sample documents, and we can just see that accuracy rate going up, you know, dramatically, particularly with these latest models. And so right now, I feel like we're in a pretty good place. I rarely, I honestly, I rarely see notebook LM just wildly hallucinate something. I mean,

one thing that's really important, people may not know this. I take this for granted because I've been living with this product. If you load in a bunch of sources about the history of NASA and then ask a question about Taylor Swift, in general, notebook LM will say, I'm sorry, Your sources don't discuss Taylor Swift, so I can't answer this question. And obviously, the model knows a lot about Taylor Swift. It's probably a pretty big fan of Taylor Swift. But it's been specifically instructed to not answer questions that are outside the source material. I mean, to a fault, I think, sometimes, you know, one would like to bring in some outside knowledge, you know, and trust but verify with that.

But we've aired on the side of, like, stick to the facts in these documents. And so with the increase in accuracy and with the UI of citations, there's more. more things we could do, we could make that fact-checking feature, you know, double-check this kind of as something. But I feel pretty good about where we are in terms of the quality side of that. Where we are kind of state-of-the-art is upload many, many documents, ask a complex question that involves, like, multiple kind of variables drawn across, like, multiple documents, get a deep, you know, long answer with citations, follow those citations to read in the

original text. Like, I think that notebook Ellen, you know, kind of does that flow as well as anybody right now. I feel like you just described like a personal Wikipedia. And I mean that as a compliment, like the thing that Wikipedia is best at is just being like a starting point to go learn about something on the internet, right? Like you open a Wikipedia page, you go click on all the references at the bottom and then you go read those references and you're off and running. And I feel like what you just described is like I can shortcut that with any process of any single thing that I want to learn more about, I just dump it in. And I'm like, what's going on here?

And it'll just be like, here's some stuff. I have so many things to say to that, David. You're going to have to give me 20 minutes. Okay, so one thing is I'm in the process of thinking about what I'm going to write the next book on. And so I created that this is just kind of second nature workflow for me now, but, you know, wouldn't have occurred to me a year ago. So I created a new notebook called The Next Book.

And whenever there's an idea that kind of comes in my mind or an article reader or something like that, I dump it into that notebook. It's the new Spark File. It's the new Spark File. Wow. But it's focused on this project of like, what should the next book be? And so the other day, you know, late night, this kind of thing I do late night because I have no life. My children have all got to college and I have nothing to do with sit around.

You just talk to Bart. It's cool. I talk to Bart. So I was like, I wonder, you know, has there been a good book written about the anti-nuclear movements, anti-nuclear power movements of the 60s and 70s? Because that's an interesting case where we like stopped a technology partially and it's tracks and maybe made a mistake and, you know, how do we interpret that? And so I just went and grabbed, like, two Wikipedia pages. In the past, I would have, like, you know, started by reading through

the Wikipedia pages, but I brought it into a notebook and I was like, I'm Stephen Johnson. I'm thinking about writing a book, you know, in the mode of my other books like Infernal Machine and Ghost Map, potentially about the anti-nuclear powers. Take a look at these. Like, what do you think of there? Are there any interesting storylines that would be good starting places? Like, what, you know, what would be good there. And it just does it. Like it goes through and it's like, well, you can focus on this period.

This figure is kind of interesting, whatever. And then I went in read all the, you know, I'm going to read the material. But as a first glance, like, inroads the material, it's amazing for that kind of exploration. The experience of navigating through a book through a conversational interface is really interesting. And it's one of these things where, like, you know,

until now, if you wanted to explore the ideas in an author's work through a conversation, you could only do that by finding the author in person or finding a scholar or a tutor who's an expert in the author's ideas. And like, that was it. There was no other way to do it. But now you can load a book in

and you can start with like, I'm interested in this, tell me about that, and then slowly read the book in a nonlinear way through a conversational interface kind of dipping in and out of the kind of original passages, which is terrible if it's a novel or terrible if it's a, you know, straight, linear kind of history book like I sometimes write. But if it's an advice book or, you know, a book of ideas, you know, there are lots of books that I think could be explored in that way that just, you know, weren't possible before. Does it feel like cheating to do that? Like, for you as an author, you have worked through

these books, you have done the hard work, you have stayed up late at night. I'm sure you've, like, woken up in the middle of the night with a book idea. Does sitting down and asking a computer what your next book should be, feel like cheating? I think if I were literally like, hey, what should my next book be? One, I don't think it would, I mean, right now, in a way, it's easy to answer this question right now, because it wouldn't be good enough. Like, it wouldn't generate, like, tell me what my book should be, and now start writing it, like, it just wouldn't be able to do it. Okay. Current trend lines continue. I was going to say, give it a minute. Come back to me in two years, we'll see. But what I have to do is, like, I'm not using it that way. I'm using it, like,

these very, like, kind of targeted queries. Like, you know, I was, I actually was using the NASA notebook because I was interested in, like, you know, maybe there's something to be done about the Apollo 13 fire. And so then it was kind of like, okay, look, I need to know what I need to read, because it's a million pages, a million words of transcripts. I actually don't need to read the whole thing. What I want to read are the sections that are relevant to the Apollo 13 fire. And so I just went to notebook and said, tell me what I should read.

You know, and it was like, you should start here. you should read there, and I can click immediately in there and read through that. And so to me, I'm using it as a way of accelerating the process of discovery that would just have been painful to do before, but I'm making just as many unplanned serendipitous discoveries along the way. I mean, I think the criticism is like, well, if the model serves it up to you so quickly, even if you end up writing the book, you've missed the surprising thing that you would have never found, you would have only found it by reading through it in an incredibly linear way. And I think that that's just not true.

It's constantly surfacing things that I hadn't thought of and making connections that I hadn't thought of. And so it's helping me understand the material more deeply. Where it gets complicated, and you and I've talked about this before, the way I think about it is it's a tool that helps you understand things. If you are genuinely interested in understanding things and having that understanding be in your brain, it's a huge net positive.

100%. Like, if you go into it with good intentions, it is an amazing tool for thought. It helps you have richer, more complex ideas, understand material better. If you are not interested in understanding things, but rather interested in creating the illusion that you understand things

and just want to bluff your way through life without ever actually understanding anything, but creating outputs that make it look like you understand things, it essentially will help you do that well. And the question is, the question is, like, how often is that useful as a strategy in the world? And to me, like, if you go into work and you're like, I've got this great hack, I never read any of the emails from my boss. I just, like, put it into the model and then I output anything. Eventually, your boss will, like, have a conversation with you and you'll say, Stephen, you don't

understand anything. You have not learned anything, and you'll get fired, right? It just doesn't, it's not a good long-term strategy. The one place where it's tricky is school, where there is potentially a strange incentive to bluff your way through something. We think that, you know, we're seeing a lot of adoption of notebook elements, 18 plus currently, so it's not usable by high schoolers, but we're seeing a lot of college and higher ed adoption of it. It's probably our biggest community so far. We are very excited about that use case, but we're also like, you know, we've done a lot of things to, to, you know, ensure that, you know, it's hard to use it to bypass understanding.

Like it's constantly steering you back towards the original text. You're always kind of being pointed back towards that source material. It's always there in the UI with you. And there are more things we can do on that front. But that's the line that I think is like, you know, if you go into this with good intentions, I don't feel concern about using the tool in this way. Okay. I buy that.

And it feels like part of the line there exists in kind of what you let people make with what is coming out of it. And I wonder, like, we've talked a lot about kind of the inputs and the processing and the understanding. And it's not, you're sort of one, you know, published this answer on Amazon as a self-published e-book with your name on it, away from this being a very different conversation. But my sense is you're deliberately or not not investing a ton in the like, write me a research paper from these six documents kind of use case. Yeah. If you look at the notebook guides, like one of them is called study guide, right? It creates a study guide. It doesn't replace your work.

It helps you actually, like, you know, it gives you a set of review questions. It creates a little multiple choice quiz. It creates, like, essay suggests questions. It creates a glossary of key terms. So, you know, that's what we're trying to do inside of the product. We're steering everybody towards that kind of, like, help me understand mode with the idea that, you know,

the people have different ways of understanding. There are different ways that people have, like, information sticks with them through different forms. And, you know, our job is to kind of do that. And that's, you know, it turns. out, I was writing about this a little bit this week, that, you know, translation and summarization is something that the models have always done quite well, like, from the beginning, like, kind of like the first deep learning breakthrough that really made a difference to consumers was really like translate, Google Translate and others, you know, that it was

basically like take this set of tokens that are in this language and turn it into the set of tokens in this language. And, you know, they're really good at summarizing in all of its different forms. And so we're, you know, we've kind of like embraced that because, you know, embrace the things that the models do well. It's generally a good strategy. All right. We have to take one more break and then we will be back with the rest of my conversation with Stephen Johnson. We'll be right back. Support for the show comes from LinkedIn. If you're a small business owner, you know that every hire counts, but time and resources are limited. Finding, connecting with, and screening the

right candidates takes up valuable time you could be giving to your customers. That's where LinkedIn Hiring Pro comes in. It's built to be your hiring partner, helping you find the right candidates faster. That way you can hire with confidence without turning it into another full-time job. Hiring pro streamlines the entire process from drafting your job to shortlisting candidates and conducting AI-powered interviews for initial screenings. Its updated conversational interface lets you to describe what you need in plain language. Nearly 60% of hires find a candidate to interview within a week. With Hiring Pro, you spend less time searching and more time connecting with the right talent.

And instead of getting buried in resumes, you get a focus shortlist that actually moves your hiring forward. Join the 2.7 million small businesses using LinkedIn to hire. Get started by posting your job for free at LinkedIn.com slash track. terms and conditions apply. We're back. Okay, we have waited long enough to talk about audio overviews. Those wild AI-generated podcasts, notebook has been working on. So that is what I asked Stephen about next.

So, and speaking of that, actually, this is a good time to talk about audio overviews in the sense of different ways that people learn. Would never in a million years have guessed this was going to be the next big feature to come out of notebook LM. So rewind the history a little bit here and tell me where did this come from? Yeah, it's a really interesting story. So let's just describe it briefly. It has been all over the Twitter sphere and other places.

I still call it Twitter. Sorry. It's okay. I do too. This is just a few days ago that it launched. So basically it's an extension of the notebook guide kind of panel. Instead of taking your sources that you've uploaded and turning them into a briefing dock, a text-based briefing dock or an FAQ, it turns them into a roughly 10-minute-long podcast

conversation between two AI hosts who discuss the material, tell stories, share interesting facts, banter in a playful way with remarkable human-like intonation and all this stuff. And so it's basically the idea is like some people like to learn through reading a briefing doc. Some people like to learn and remember better when they hear information conveyed to an interesting, stimulating, engaging conversation between two people. And you couldn't create that artificially before. Now you can. So we've added it to the suite of tools. That feels like it causes a thousand new interesting things. Like on the one hand, it's, it's not the thing that you were describing earlier in this conversation, which is something that

is not full of personality. It doesn't say I a lot. It's not, this is not a tool anymore. This is two slightly weird people hanging out in my ears for 10 minutes. And they are like, whatever dial you have that says like make them say puns and try and be kind of cringly. It's like that dial is at 14 out of 10 on most of the ones that I've listened to. And I kind of get why, right? Like if you take the it should have no personality, it should just be very straightforward approach that makes sense on showing me a bunch of text and apply it to a podcast. That's a bad podcast. But it does feel kind of incongruous with a lot of the stuff you've been saying is like core to what you've been trying to do well.

I mean, it's such a great question. So let's just play. I want you to play this case people haven't heard it. Okay, so get this. Picture a world where the hottest tech isn't microchips or the internet, but dynamite. Yeah, dynamite. And it's not just blowing stuff up. Think skyscrapers, railroads, massive engineering projects.

But also, you guessed it, assassination attempts, terrorist plots, all that volatile stuff. It's a wild period, right? Wild is an understatement. And that's where our deep dive today comes in. We're cracking open Stephen Johnson's The Infernal Machine, a book that digs into this explosive era where dynamite, anarchists and the birth of, believe it or not, modern data collection, all collided. Yeah, get ready to have your mind blown because the connections he makes are mind blowing. Absolutely. So it's just everything after, yeah, dynamite just kills me.

I love it. You know, one of the things is it's they are enthusiastic. And so people have been uploading their CVs. And it's just like, if they were feeling down about yourself, like, they're like, oh, John Smith. Look at his, make you a promo podcast. It's Melma cum laude from. So I think when we first talked about what was then Project Tailwind, for the verge, like, you know, Verizon and I were talking about how, you know, we don't, the model doesn't have a persona, really. Like it doesn't use the subjective eye in general.

It's not trying to be your friend. It's just trying to get you the information you need. And that was that was kind of the house style that we felt was appropriate for what we were trying to build. But if you want to have a conversational audio format, there is no way to do that without they're having a sense of like human personas. It just won't work. No one wants to listen to two series talk to each other, right? That is not, I'm sure. There's a lot of that on the internet and it's not good.

The new series is better, but I'm sure, but the old series, right? You know, you want to listen to two robots talking to each other. And initially it was another team inside of labs that had built this prototype that was like, you know, create a document. There are actually a couple of projects like this at Google. There's another wonderful one called Illuminate that has a slightly more like kind of scholarly tone to it. But it was just one of these things where like it became possible to do this. And so, you know, some folks were exploring this. And it was a very good demo.

We've seen from the last few days with audio overviews, like, people are impressed, people want to share it. But it didn't kind of have a place to live. And so right before I.O., actually, there was this idea of, like, well, maybe we could actually put it into, you know, would it make sense inside of notebook? And, you know, we were just rolling out the notebook guides. And when I first heard it, the podcast stuff was like actually even bigger, like the host had names and things like that. Okay. And so, you know, but I like, it definitely may, you know, I think Verizon, I immediately saw that it was a continuation of the philosophy of the guides that, you know, we'll take whatever you give us and we will turn it into the format that makes it easiest for you to understand. And we know from, you know, the success of podcasts in general that people do like to learn that way.

to travel with audio and drive and, you know, walk around the city listening to it. So we knew there were a lot of reasons why audio conversations would make sense. Are you starting to have similar conversations about things like video? I can just imagine, like, I'm starting to think about what notebook LMLM looks like when you're trying to be like TikTok native and YouTube native and Instagram Reels native. It's like, it's just what a wild kind of road to end up heading down in terms of like, how do we communicate to people where they are, but sort of in this house style. This is one of the reasons why it's good that I am like 56-year-old gray-haired Stephen is not actually driving things like because I wouldn't be terrible at helping with that.

But like, yes, I think that could be part of our future. One of the things I think is really interesting about it in the response, it takes like four or five minutes to sometimes three to five minutes to generate one of these. And that's because there is a really complex series of kind of, you know, compute and inferon it's going on. Basically, you can think about it kind of in an editorial way, right? It's kind of drafting a version of it, and then it's filling in the details

of the script, and then it's revising the script based on the overall goals. There are multiple cycles of basically an edit cycle, and that takes time. And then crucially, there's a stage where, my favorite new word is

where disfluencies are added, right, so that it deliberately makes it kind of noisier and more like the way that people talk when they're in conversation, where they overlap and they have partial phrases, they complete each other's thoughts and stuff like that. All the things that make a traditional transcript hard to read

if it's not been cleaned up, if you don't have those things in, it's on the wrong side of the Uncanny Valley, or it's right in the middle of the Uncanny Valley, I guess. You can debate, and obviously, like, we're going to, you can imagine we would introduce abilities to steer in different directions and dial up, dial up, down things, or focus on different topics.

But the basic structure of kind of, you know, doing this long edit cycle and then humanizing. Yeah. The thing I called wrong on this one was that I thought audio overviews would be one of those things that people were like, oh, well, this is silly, right? It's like every time a new image generator comes out, a bunch of people make wild stuff with it for 48 hours and then kind of move on and we don't talk about it anymore until some new image generator comes out, this one doesn't seem to have gone like that.

And I think it, somewhere in between the, like, is it remarkable that it's possible versus, is this actually useful? I think it is more in the this is actually useful camp for people already than I expected it to be. Like, it's definitely remarkable that it is possible. And you showed a sort of sincerely new thing you can do with AI. And whenever that happens, people get really excited on Twitter, right? Like, that's just one way to get people really rout up on Twitter for 12 hours. But I think there was, you, you overlapped the, this actually does something. thing for people side of things more than I expected when I first heard you can make an AI podcast

out of your notes. Well, I want to thank you, one, for changing your mind. But two, you remember how Jobs used to have that kind of Apple is the intersection of liberal arts and technology, whatever? Like, what you just described, the intersection of what is newly possible and what is genuinely useful, like, that's what we're trying to do at notebook. And that's kind of what Labs is trying to do, right? Like, it's just like, what can we do and what would actually be like, you know, what was unthinkable six months ago and then what would, would it be?

And, you know, part of it is, you know, I think in some cases people are using it like, I want to study and this is a better way for me to, you know, take my documents and, you know, just learn or learn on the go. The other thing is, like, creating things that are effectively podcasts where no podcasts would possibly ever exist in the real world, right? So people are like, here's this very obscure niche topic that no one, like the economics of the podcast business will never support a podcast. I want the podcast on my D&D game that I play with my friends. Yeah, yeah, yeah, no, totally. And or it's like, you know, we've seen people doing like, again, thinking about notebook element in a kind of a team context at work. Like, okay, like the week in review. Like give it all your documents.

I always say to people, one of the best ways to get to know notebook is to, if you're a Google, docs, slides, drive user, just grab, creating a new notebook and grab the last, like, 20 docs that you've created and just create a notebook with that. And then just ask you questions. And it's ability to kind of grasp, like, what you're working on and the issues and things like that. Do that and then create an audio overview.

And it's like, this is what Stephen was working on for the past week. You know, here's too happy and enthusiastic people to discuss, like, the things you've been working on. And sharing that with your team, like, it's pretty nice. Like, that's a great way to, like, look back on the week and think about what you've been working on. And, you know, you can imagine other ways to explore that as well. All right. A couple more questions.

I know I've kept up to you a long time, but I'll leave you on here. I can talk to you about this forever. Yeah, you're even better than talking to Bard. This is great. Anytime. I'm here for you. I don't like skiing either. So, like, do you think there's a, like, big mainstream use case for something like notebook L.M?

Obviously, there are. people and industries and jobs and schools. Like, I can imagine a bunch of people for whom this is useful. Do you think there is like an everyday everybody use case for something like this? Everybody every day? I don't know. I mean, to be, like, let me put this in slightly different context, right? Google is a company that is famous for making things that a billion people use.

A billion people use. Is there a billion notebook LM users out there? I think if you define it as there is an AI that is an expert, in the information that is really relevant to you, that you've curated, and that you can engage with in different forms, whether it's chat or listening to an audio overview, and you can basically cultivate your own personal AI

with a lot of information. Maybe it's all your journals. Maybe it's your company's history, whatever it is. And just by like dragging and dropping files in there, it develops this kind of knowledge of everything that has happened to you or your organization and is able to kind of deliver advice or help you make decisions or just recall a fact that you're trying to remember and create these new documents. Is that a big, like, will maybe a billion people like be doing some variation of that in five years? I think

that's pretty plausible because there's so many different things you could do in that kind of context. You know, it could be like a writer working on a book, but it could be an executive trying to make a complicated decision, or it could just be somebody trying to remember things that happen in their lives. And it's like just keeping a journal only you've got an AI that's helping you do it. So I think there's a big market for it. Okay. You know what I've been thinking about, as you were saying, that there's this app called My Mind. Have you ever seen it? Have you ever seen it a little bit? Yeah. It's a great app. You should play with it. I think you of all people would enjoy some of the

AI stuff they're doing. But I've started using that app and I've gotten in the habit of basically every time I encounter something I like, I just put it there. Like if a podcast or a video or a thing I read or a photo I take or a quote that I see, I'm just pouring it all in there with no organization, no, nothing.

And there's nothing particularly special about that app. It's just like, it's pretty to look at, so I like using it. And already, just the thing where you open it up and it has a bunch of modes

where you can just type in like movies and it'll show you all the movies and movie adjacent stuff that you've been saving. And it's like, there's something really powerful about like, here is just a compendium of stuff that I find interesting.

And there's just enough manual work in that that I think it's, there's something to solve in the like, how do you collect that data from people in a way that is both sort of easy and frictionless, but also not like gross and privacy invading. That is like the eternal Google question. I have high hopes that somebody will figure it out eventually. That is a bias that I've always had probably to a fault that, you know, like we don't, we really don't have, we don't even have kind of folders for your sources or your notes

or things like that, like basic kind of low-level things that we will no doubt add, but we've been focused on these kind of cooler features. But part of me is like the beauty, like this, the complex systems of organization where you're tagging things and putting them into the right folder and stuff like that are not

as necessary now because the AI does that and makes the connections for you and finds the thing that you're looking for. So just have one place, make it easy to grab as much stuff from as many different places and just dump it into a notebook. And then we'll do the magic and figuring out the connections or finding the information you want after that.

Again, you said it correctly before. Like, the closest thing to this before was Wikipedia. Like, I start here in this Wikipedia page about elephants and then I follow it following these interesting paths. But a dialogue is just an even better way to do that. And if it's trustworthy. And so the mode that I really... love is, and again, this is only possible with conversation history and long context, is to go on one of those, like, exploratory conversations through an idea.

And then at the end, you're like, okay, this has been great. Will you format that all as a single document that just has kind of key takeaways and insights from it so I can just capture it for later? Because I don't necessarily want to read the whole conversation again, but I want to get the, like, great pieces from it. And boom, it does it, and you save that, and that's your kind of record of what you just take. Like that, I mean, like, that's a beautiful way to walk through information space and full of surprise and unexpected turns and discovery. So that feels like a keeper. What's the next step in getting better at that? Like, one thing that it seems to me that we need pretty badly in that way of thinking,

which I love. Like, I love the idea of just like, I'm going to sit here and spend three hours learning about something. And then at the end, you're going to deliver me like a handy summary of everything that I've learned. Yeah. Amazing. That's the dream. It does strike me that one thing we desperately need all of these tools to get better at to pull that off is multimedia. Right? Like, I want to know at the end of it, like, go listen to these three podcasts and this YouTuber, you're going to love him, go check it out. And this person on Instagram is doing out. There's like, the internet is such a messy place in the best way now that it feels like that's one thing. And I know a lot of folks are working on that. Are there other things that you look at and, you know, you're working on this stuff too? Is there a next kind of turn coming that's going to make all that stuff even?

better. Right now, notebook's ability to help you discover information to put in your notebook is exactly zero. It does nothing in that. You are on your own. You need to supply the sources. We will not help you discover anything across the internet in any form. It turns out, just as a coincidence, that Google is really good at that stuff. Like, I did not know. I learned when I got there, they have this search thing that is apparently a lot of people used. So it's obviously like a place where, you know, I would love it. I wanted to be source grounded, but I would love you to stay in notebook to be able to find things.

And, you know, we would be really interested in exploring that. I think you'll see that in 2025 for sure. All right. That is it for the Vergecast today. Thanks again to Stephen for being on the show. And thank you, as always, for listening. There's lots more on everything we talked about at theverge.com. I'll put some links in the show notes to some of the stuff Stephen has written about this too.

His new book is really great. Go to Theverge.com, lots of show notes, lots of notebook coverage, all kinds of good stuff. As always, if you have thoughts, questions, feelings, or other ideas for AI generated podcasts, you can always email us at Vergecast at theverge.com. Or call the hotline, 866 Verge11. We love hearing from you. Want to hear all your thoughts. Send them all.

Can't wait to hear it. This show is produced by Liam James, Wilpore, and Eric Gomez. Vergecast is Verge production and part of the Vox Media Podcast Network. We'll be back with our regularly scheduled programming on Tuesday and Friday this week. We have a lot of headsets in particular to talk about this week.

So get ready. We'll see you then. Rock and roll.