The Science of Everything Podcast - Special Episode: Language in Brains and Machines

Episode Date: September 28, 2022

A special interview from the Padverb Podcast, in which I discuss my research on language in the brain and thoughts on AI. The discussion covers topics such as backpropagation, how humans acquire language and concepts, how artificial neural networks differ from biological neural networks, and some thoughts on the future impact of artificial intelligence. Near the end of the conversation, we also discuss some of my work on exploring the causes of economic growth and development. Link to the Padverb Podcast: The Padverb Podcast with KMO on Apple Podcasts If you enjoyed the podcast please consider supporting the show by making a PayPal donation or becoming a Patreon supporter. https://www.patreon.com/jamesfodor https://www.paypal.me/ScienceofEverything

You're listening to a special episode of The Science of Everything podcast. I'm your host, James Fodor. In this episode, I have included an interview that I did with a fellow science podcaster, the Padverb podcast. And in this interview, I talk about some of my research that I'm doing in my PhD, looking at neural network models of language and how we can represent linguistic concepts and semantic information in neural networks. And that language is represented in the brain and in the human mind is quite different from the way that contemporary language models in artificial intelligence systems. So there's some extended discussions about that. We talk about back propagation, GPT3, chatbots, and other such things. And at the end of the episode, we talk a little bit about, we switch gears

a little bit and talk about some of the issues that I discuss in my economics growth podcast series episodes. So I thought this might be of interest. Check it out if you are interested in those sort of topics. Otherwise, feel free to skip this one and wait until the next episode comes out. But without further ado, I will pass over to the original recording. This is the Padverb podcast. I'm your host, KMO, and I am speaking with fellow podcaster James Fodor. James, welcome to the Padverb podcast. It's a pleasure to be here. Thanks for having me. So I am very impressed by your podcast. It's called The Science of Everything. I have listened to several episodes. I will admit that I have kept my, I've mostly stuck to financial episodes, although I know

you cover a wide, wide variety of subject matter. And they're mostly solo shows, which, you know, a conversation like we're having, which gets turned into a podcast is much easier to do than you sitting down at the microphone by yourself and turning out a podcast, I know. So tell me a bit about the podcast, the history of it, and just the role that it plays in your larger project of being James Fodor. Yeah, sure. So the podcast I started, I think, 12 years ago now. And the reason I originally started was just because although there were a lot of science podcasts available, and I guess there still are, I didn't find very many that were focused on just sort of, I guess, explaining science content in a sort of a straightforward direct way, kind of like lectures, I suppose, but tailored

to an audio format and in a bit more accessible way than, you know, average sort of university lectures. So, I mean, that was the sort of content I was interested in just because I wanted to learn more stuff about different areas of science. And so that was one of the reasons why I decided to start the show to just provide something a bit different that was not really available very much. And I still, I don't think there's still very much that's in this area. There are a few history podcasts that kind of do this. But I guess in history, it's perhaps a little easier because of the narrative form that the material often takes. So I try to bring a bit more of a narrative flavor to the science that I give that I deliver. And the other reason that I

started, well, one of the other reasons that I started to show is because I find it helpful to kind of focus my own reading and research on a particular output, like doing something in particular with it. And being able to sort of synthesize what I've been reading or thinking about in a, you know, hour-ish long format is kind of helpful for me. So that's sort of a, a selfish reason why I started the show. And I've always had an interest in many different areas of science, so I wanted to keep it fairly general. So that's why I call it the Science of Everything podcast. And I try to mix around the topics so that, you know, I can cover a wide range of topics. And I think also that the aspect of trying to provide a unified framework for thinking about

science is something that I also find valuable. Well, you've been doing it, you say, for about 12 years. there's 130 episodes. So what sort of production schedule do you keep? Well, it's varied quite a bit over that, over that time, depending on my commitments. At the moment, I try to get an episode out approximately once a month. That does vary a little bit, depending on what I'm doing, because I have other commitments as well. I'm doing a PhD and some other things.

But yeah, roughly once a month. And so I will have to spend one or two weeks reading and bringing materials together. and then when that's sort of at a ready state, I'll record the show and then editing and then put it up. So it usually takes like a weekend to record and edit and finalize all that. And so I'll need like a week or so before that to prepare or maybe two if it's a longer show. So it sort of works out about once a month. Yeah, the impression that I get listening to it is that most of the work is in the research. Yeah, absolutely. Yeah, the recording doesn't take too long once the notes are all prepared. I don't script the show. I find that that adds to the, well, it's not exactly a conversation. It's more of a

monologue, but it adds to the more conversational flavor to it. But I do have a set of, a detailed set of notes that I work from. And that's, yeah, preparing those is what takes most of the time. Well, you sound very organized. It's clear that you know what you're going to be talking about next, but, you know, sometimes you'll pause. It seems like you're searching for words. It doesn't sound like you're reading a prepared script. So are you working from an outline or how much, how finished is the show when you sit down to actually start speaking into a microphone? Yeah, so I have an outline set of notes that I'm working from, so I know what I want to talk about, but I haven't, as I said, I don't script out the exact wording, and that's deliberate because I prefer that style of delivery essentially,

and I want to replicate a style of essentially a lecture where the material is prepared, but the exact wording isn't. And yeah, so sometimes when I'm recording a show, I will find that there's a gap in what I wanted to explain. So I'll sort of pause that and adjust the notes and then and then record and then edit that in. But usually I record it mostly in one go and then just edit disfluencies or bits and pieces in and that takes another couple of hours. But yeah, I find that works fairly well. And also scripting scripting word for word would take a lot longer.

And I think it would also mean that the delivery would be flatter, which I don't like. So yeah, that's sort of why I do it this way. Yeah, if you're going to read a word for word prepared script, you basically have to be a voice actor. Otherwise, it is going to sound pretty dull and repetitive. Yeah, yeah, I think so. Yeah, I think it gives a different flavor to the show as well because most shows are either fully scripted or a conversation. And this is supposed to be sort of partway in between that. Well, it's a conversation of one.

Yeah, yeah. Do you ever do interviews? I have a few interviews, like this one. Occasionally, I will have someone who wants to collaborate or would like to talk about some particular topics. I've put a few of those up. I've reviewed one or two books on there that are relevant. I release those as special episodes just to sort of give something extra.

But the main line of the numbered series of podcast is just entirely me. And I intend to keeping it that way. I also want the podcast to be evergreen content because it's mostly focused, well, it's pretty exclusively focused on established scientific knowledge. So a few of the episodes of the older ones may do with a bit of updating here or there, but for the most part, it's still current. And that's also one of the things that I wanted to be a bit different because there are so many science podcasts that focus on science news.

And personally, I'm not really a fan of that model because I don't actually think that science fits a news model very well. Science makes progress with very gradual piecemeal discoveries or developments, many of which turn out to be wrong or only slight advancements on current knowledge. So I don't think that fits a news model very well. And I'm kind of critical of a lot of science journalism for that reason. So I try to focus on things that are kind of established or if it's not known, then I'll explain what's not known and whether the gaps lie. And for the most part, that doesn't really change. So that's why I have a series where I try to just progressively build on the topics that I've talked about before. Well, when you put as much effort into creating a podcast as you do, you don't want it to basically get stale because it's no longer relevant to the news cycle. Yeah, exactly, exactly. You mentioned that you're working on a PhD. Tell me about that.

Yeah, so I am studying at the University of Melbourne in Melbourne, which is where I live, and my topic is broadly cognitive science. Specifically, I'm studying how humans represent semantic knowledge. So that's like knowledge about word meaning and how we store ideas and also combine different ideas together. So every day we're faced with novel sentences and phrases that we've never heard before. And somehow we have this ability to put different ideas together and form an understanding of what people are telling us. So it's not like we have this server rack in our brain where we've just sort of memorized all of the sentences that people tell us that that's not possible. So we have an ability to synthesize ideas together.

And that's kind of the topic that I'm particularly focused on. And my research specifically focuses on trying to compare artificial intelligence models of word representation and word meaning and language representation with how humans think about and represent language. Well, it just so happens that I've been interviewing a lot of people about artificial intelligence. And in particular, neural networks and deep learning and training very large language models on enormous data sets. Oh, cool. And so if you read a popular science article, on neural nets, you're likely to read a claim that, you know, these things learn like humans do. And I know that they don't. And you know that they don't.

But why is that such a silly thing to say? Yeah, it surprises me that this, I mean, this is one of the challenges of science journalism I think before, among other things. But, I mean, there's a number of major differences. I think one of the biggest differences is that the models, the artificial intelligence models that we have learn by essentially reading. huge corpus of text, so just the whole internet essentially and digitized books and other things as well. And extracting, extracting statistical associations from the text that they're processing.

The amount of data that they process is, I mean, I've done a back of the envelope calculation at least 100 times more than a human could possibly read like in their entire life if they just like read all the time. And so realistically, it's probably a thousand or more times as much. And so, I mean, it's clear that humans don't learn language by just extracting statistics from huge corpus of text. That's one issue. Another issue is that the specific algorithms that the machine learning architectures use are not really plausible. So many of them use a technique where they pass the data through a very large neural network with many connections and the network is trained to predict the next word in the sentence or often it's the next sentence or maybe it's

masked words within the sentence. It varies between the specific models. But that requires essentially comparing a prediction against an error signal. So like you have the true word or sentence or whatever it is and then the prediction and then you compare this against each other. And then there's a technical back propagation where those errors are propagated back through a very big network, right? And this sort of algorithm is not really plausible in the brain. Both the sort of prediction error thing, which is unclear, but also the back propagating through many layers is definitely not plausible. It may be that humans do something vaguely like this, but the specifics are definitely not how it works in the brain. And honestly, we don't even really know how it does work in the brain

other than some very general points. So, yeah, the size of the data that's used and the way that it's processed are fundamentally different in humans compared to the machine learning models. And that's one of the things that I'm interested in in my research is because I think we can learn some things from these models, but it's the sort of question is exactly what and how they relate to, how we process things in the human brain. And I think that's a very interesting area of research at the moment, trying to nail that down a bit. I'm about to ask you a question that if I were being interviewed and somebody asked me this question, I would be uncomfortable. Go ahead.

You brought up the word. What is back propagation? How does it work? And why is it important on the topic of artificial intelligence and neural nets and deep learning? Oh, sure. Right. So I wasn't intending to explain the details, but I'm happy to. So I mentioned it because back propagation is an algorithm.

them. So it's just a series of steps that the computer programs use that's used in a very large number, probably really all of the major like machine learning models that have been developed, particularly the language models. So it's really what's given rise. It's one of the things that's given rise to the kind of deep learning revolution we've seen in recent years, that and more computing power. But what it means is effectively, for those who know, it's an application of the chain rule from calculus, but don't worry if you don't know what that means. The important point is that it's a way of assigning credit to particular nodes in a network, right? So suppose I have a big network that has many different nodes connected to each other and I'm trying to predict like the next word in a sentence,

right? And suppose that I get the prediction wrong, right? Well, what I want to do is I want to update the weights that connect these notes together so that next time I'm more likely to predict the correct answer. That's how these networks are trained. But the question is, how do I assign credit to particular nodes? Like, how do I know which weights to update and which ones to leave the same or which ones to increase, which ones to decrease, that sort of thing. Back propagation is an algorithm that allows a credit assignment for each of the different layers backwards across the network. So if I have like 10 different layers connected to each other, I need to send the errors

back all the way through the network to the beginning so I can update all of the weights. Otherwise, I'll only update the weights at the very end of the network, and that's not very helpful, right, because I have many, many layers connected to each other. So back propagation is the specific algorithm that tells the weights in the network how to update and how to assign credit for which nodes were, which connections were important for making that decision and which ones weren't important. And without that,

we wouldn't be able to train these big neural networks that are currently used in these language models and other applications. And the whole point about that is that this algorithm is, it's pretty much known to be not plausible for the brain to implement, because the brain doesn't have access to these signals that sort of travel backwards across a network like this.

There are a whole range of different projections that exist in the brain, but none of them seem to be able to carry this particular type of signal. So one big question currently in neuroscience is, does the brain implement this kind of supervised learning where you have the labels and then we can tell whether a prediction is sort of correct or incorrect? And if so, how does it assign credit to like which awaits to update? And is there anything like that propagation or some approximation of that that happens in the brain or not?

At the moment, there's no evidence at all that anything like this happens in the brain. So anyone claiming that these artificial neural network models are doing something like what happens in the brain, that's speculation. Like it's possible that something approximating it happens in the brain, but no one knows if that's true. Well, I think even if it is discovered tomorrow that there is some sort of back propagation process going on in the human brain, and I don't think that's going to happen, but, you know, if it does, still humans learn very differently from machines in that humans are, it seems

anyway, primed to learn a language very early, and they don't need nearly as much data, you know, as machines do. And they're not just making statistical correlations. They're acquiring concepts, you know, and associating them with sounds. And as far as we know, machines don't acquire concepts. They just make statistical correlations between groups of words. I'll stop there and let you correct anything I may have gotten wrong and expand on that. Yeah, that's correct. And that's something I wrote about in an article as well, where I discussed this recently. So the, I mean, that's effectively what my research has focuses on human concepts. What exactly is a human concept is pretty hotly debated. But I think one way to think about this is that it's an abstract representation

of something in the world or some construction that we have that's an abstraction. So some concepts are about very concrete things like tables, chairs, dogs, whatever. Others are more abstract, like democracy or freedom or whatever else, right? But the idea is that there's some sort of stable representation or mostly stable representation in our brain somewhere that, that forms a representation of this idea, this thing in the world, that we can then activate or bring to recollection when we need to think about or reason about that the corresponding idea. So do neural networks have concepts? Well, in a very vague sense, they have some kind of representation of a word say, but it seems that they're not really anything like human concepts.

One important difference is that the language models only have text. Well, most of them only learn from text, right? So really what they're doing is making big associations between when certain words appear in texts and when other words appear in text. So they're making all these complicated correlations. And humans actually don't learn language from text. No one learns to speak by reading that we learn by hearing and interacting and speaking ourselves. And a big part of that is the interaction part. So pointing at things in the world, interacting with things in the world, feeling emotions,

corresponding with things, you know, seeing a face that corresponds with, you know, mom or dad or whatever. That's how we learn. And so the technical term for that is embodiment, right? So language is embodied in the way we interact with the world and the way we perceive and so forth. And the current models don't have any of that, right? All they have is the correlations, whereas those sort of linguistic correlations probably play some role in human language, but I would say only a small role because of all of the other things that we know are more important, especially when you're talking about children learning

language. And so as a result of that, it seems that the language models don't learn the same type of a richly structured set of concepts that humans do. And by that, I mean that humans have a whole host of a concept is not like just a label that you apply. Like I see a dog or like, oh, a dog. Concepts are much richer than that because we can use them to reason. So like if I ask you, how many legs does a dog have or does it have fur or, I don't know,

could a dog fit inside a letterbox or random questions, right? You could probably answer many of these, even though you've never thought about them before, right? A classic example that sometimes is given in the literature is that machine learning models don't pick up facts that a lot of humans know but don't talk about very often. So it turns out that often we don't describe bananas as yellow because everyone knows that bananas are typically yellow, right? And so if they're not described like that very frequently, then language models won't pick that up because they just pick up word correlations effectively. Whereas humans will

just learn those sorts of things and be able to reason about them in a very novel way. Language models can do a little bit of that, but the abilities are much more constrained compared to what humans can do. And it seems that a big reason is they just don't have the same type of interconnected, richly structured, flexible concepts that humans do. They have a very kind of impoverished version, which is mostly just about word correlations. So I'm referencing a comment in the chat here. I'm not going to read it verbatim. I'm going to paraphrase it. But Parker asks something to the effect of, why is it important if machines don't learn the way a human brain does? Well, that depends on the application, right? So if we want to develop language models

for specific purposes, like I don't know, customer service, or for use in game applications for creating dialogue or for automatically moderating comments or emails, things like that. There's lots of applications for these things where it might not matter at all whether these things form concepts or learn language the way that humans do. But there are a number of cases where it probably does matter. So one is from a scientific interest, which is sort of my focus, if we want to use these models as scientific models of human language learning and or human concept representation. Then obviously we want them to approximate the way that humans do it because that's sort of

the purpose of a scientific model. And anyone making claims about these models as if they are like scientific models of how humans actually reason and perform language tasks, then that difference is going to be relevant. But the other thing is that for a lot of more sophisticated applications, I mean, we sort of want these models to be able to reason in a more. human natural human way. So, I mean, one example could be if we want to use a machine learning algorithm for like moderating Facebook or YouTube or something like that. I mean, and they are used for that purpose already, right? But there's still a lot of things that the models can't pick up. And one of them

is sort of sensitivity to context, right? And the nuances of like one sentence following on from another and picking up something. Because a lot of the models have a limited, a limited like contextual window. So they just forget everything that's more than a certain number of words previously. And part of the problem there is they don't really have, they don't really have like a stock of world knowledge that they can appeal to that then is access in a context specific way. So if I'm talking to you about my podcast, then we know that that's the context, right? And I can remember something I said, you know, 10 minutes ago, hopefully. Whereas, you know, the machine learning models don't really have anything like that, right?

So, and plus, you know, they can't do the same sort of flexible inferences and so on that we just talked about. So in some cases, that's not going to matter, but in other cases, they're going to miss things. And often they'll just sort of pick up on key words or phrases that are associated with other things and not deliver the result that a human would. Right. And so another example actually that comes to mind that people have worked on is automated marking of essays or test answers. That would actually be really useful for education if that could be developed to a point where it was as good as humans. But again, that requires a sophisticated, like being able to mark an essay, for example, requires an ability to synthesize. the argument that's been made and follow the steps of like how that's been argued out. And you can use one thing that these models are very good at is picking up on like grammar and word use, right? So you can use these to mark an essay just based on like, is it grammatical and does it use longer words and things like that, right?

And you can then build a dataset where it's like, well, this essay looks like the language that's used in high scoring essays. So you could potentially be accurate there. But the model wouldn't really have a clue as to whether the argument that it was making made any sense. or the points logically flowed and so on. So again, if you wanted to use it for that application, it would really have to be more sophisticated than current models.

So I guess the point is that for more sophisticated applications, you really want a model that is able to think more like a human does. And for that, I think we need to move beyond the current techniques that are used. What I've noticed is that language models, they don't even behave like, say, you know, computers from science fiction of yesteryear did, and that they're not very logical. You know, they can't construct good arguments.

They can't dissect arguments. They're just not very good at reasoning. All they're really good at is searching for language which seems like it might plausibly fit into this context. Yes, well, it's one of the challenges here is that because these models have read like the whole internet, anything that someone has said on the internet could potentially find its way in the model. Now, sorry, you wanted to comment? Yeah, I was just going to say, all the crap that's on the internet, all the nonsense and just gibberish, is available to affect the behavior, you know, the language behavior of these models.

Yeah, so it's not the case that the models just memorize the whole internet, but it is the case that the weights that they've learned that shape their performance are controlled based on everything that they've read. So if there is a particular combination of words in a certain context that they've sort of updated weights on, they could be reproducing that in response to anything that you'll ask it. So this is sort of hard to test, right? But this is one caution that I always make when people report on apparently like creative or insightful behavior in these models is that, well, how do you know that it hasn't learned that from somewhere on the internet?

You'd have to sort of verify that that wasn't part of its training set or that it didn't affect its weights in some way. I don't actually know that there's been too much work into that. It's kind of hard to find if something's not on the internet, I suppose. Which isn't, I'm not saying that, again, these models just memorize things. They are more sophisticated than that. But because they are trained on such huge datasets, it's very hard to tell what they have and have not seen. And that comes to your point about being logical, is that they're not trained to be logical, right? You can construct algorithms that will logically process arguments, but these aren't that.

They're trained to replicate patterns of text that they have seen. And so very often they will be able to provide arguments if you ask them for things. They'll have a conversation with you. But quickly, they'll forget about what they've argued. They very often make inconsistent arguments. I guess maybe that's more human, like, that we'd like to think, and get into loops that they just go around in circles or say things that just don't make a whole lot of sense. So yeah, I agree that they're not very good with arguments. And that shouldn't really come as a

surprise because they haven't really been trained for that. They've been trained to replicate speech that sort of fits with surrounding speech. And so, yeah, they'll say things that kind of look like they're an argument in isolation often. But then when you look at the whole context of the discussion, that it often kind of falls apart. At least that's been my experience. There are many very amusing YouTube videos of GPT instances talking to one another, basically carrying on conversations and going in very strange and unexpected directions. But what strikes me is that they can't start the conversation. There's got to be a human to type in a seed phrase or something to get things going because they just don't have an agenda, really, other than predict what comes next.

Yes. I mean, I suppose you could put a random seed in, but most of those conversations probably wouldn't be very interesting. But yeah, I mean, that's the other thing, that language serves a purpose. It serves a communicative purpose for humans. Of course, sometimes we say things without having a particular reason, but in general, there's some purpose behind what we say, that we have an intention that we carry out through language. Language is a way that we express ourselves, that we try to get other people to do things, that we convey ideas. And that all sits inside a range of other cognitive capabilities that we have, like our memory, perception, emotions, you know, social needs and things like that. And these language models don't have any of

those things. They just have the language associations. So, I mean, there's no real sense that the models want to say anything or have any reason to say anything. That's just not part of what their model to do. And we shouldn't really expect that. But the idea that we can, the idea that you can model language through a series of word or sentence associations external to any communicative purpose is a bit strange. And again, for some purposes, that might be sufficient. But I think that, Yeah, we should be really reticent about interpreting the results of these models as if they have any communicative purpose, even though they may seem to depending on the particular example, because sometimes you do get surprising results. It's like, oh, that's interesting.

I didn't expect that. But they're not modeled to have a communicative purpose. And because that's so central to humans, I think that that's another central element that's missing. So my academic background is far behind me. I went to college in the 90s. And I did graduate study in the philosophy of science and the philosophy of minds. So I was, you know, thinking and reading and writing papers on these topics. Yeah. You know, back like 1995, 96, you know, long before the crucial year of 2012 and Gregory Hinton's team and, you know, all of the, the breakthroughs that led to this flood of new applications and excitement and research now. And I can't claim that I've been working at this for, for decades. You know, I did it for a few

years decades ago. So I can't be as indignant as some of the people. But I, I, I can't be as indignant as some of the people, but I'm going to channel the indignant attitude that I've heard from other people of, look, philosophers of mind have been trying to work out in the abstract, what constitutes a mind, what a mind has to do, all the different tasks of, you know, taking in data from the world, synthesizing it, figuring out what's relevant, and then directing the body into action or saying words or, you know, making use of that information, creating these counterfactuals, factual fantasies about what might happen if we do this or if something else were the case other than what we see, you know, what would be different. All of this stuff is going on in the mind in real time such that you can say something to me.

And with no particular lag, I can say something back, which demonstrates to you that I understood what you meant. And what's more, I've thought about some portion of it that maybe you haven't. And I'm suggesting a new avenue for inquiry. people have been thinking about how a mind has to be structured and organized for decades. And then you get, you know, these young bupps who haven't done the work, you know, who haven't earned their stripes who come in and think that they can just build this, you know, multi-level neural architecture and, you know, apply a back propagation algorithm to it and feed it a notion of data.

And they don't need to know a thing about the mind. They don't, oh, you know, all of the previous research, all of the previous work, all of the previous study and just painstaking and, you know, defending one's ideas against challenge from peers, all of that is meaningless. All you need is a lot of data, you know, a particular architecture and a back propagation algorithm and, you know, call it quits. It's done, you know. So I imagine you've encountered some version of that indignant attitude that I've just articulated. Yeah, there are, There are certainly people who hold the view that human, well, human cognition more generally in language specifically may just be a very large neural network with lots of data.

Typically, they come from the machine learning space. Then there are also on the other side, particularly, I think philosophers. So Noon Tomsky is an example of this. And my namesake, Jerry Fodor, probably would be because he was when he was alive if he was around to see the most recent examples. And they say basically that these machine learning models tell us nothing at all about language or the mind and that they're just complicated parrots essentially. So, I mean, cognitive science has been pretty controversial from way back in the 50s when it started,

and I guess it still is, right? I mean, I sort of sit somewhere in the middle of these two exchanges. I think that the models do tell us something about language. One thing is that it's surprised many people how good the performance of the models is in replicating human speech, even with a, in a sense, fairly simple architecture. Actually, one thing that I was playing with for a subject, I guess a couple of months ago, was something called Ngram language models. And these are kind of old school models that aren't really used anymore,

but they date back to the time before the deep neural networks. And they're actually incredibly dumb, right? It's basically just you process a whole bunch of text and count the number of times each word appears. But then you also count the number of times, like each combination of two words appears and three words and so on. and then if that combination doesn't appear, then you make an estimate on how often you'd expect it to appear based on other correlations. So it's literally just counting. It's even much simpler than the current deep learning models.

Now, what's interesting about these n-gram models is that when you get to the higher like five grams and so on models, they can actually produce fairly human-like speech, even though that there's no syntax or anything that's built into the models, right? And I want to emphasize that they're incredibly dumb. They only just count the number of times different patterns of words appears. So I think that it's interesting how you can get very human-like behavior with, in a sense, a lot of data and just really, really dumb models. But also maybe how humans are very ready to perceive intelligence and kind of content behind language. Because that's what we're used to doing, right? In the real world, we don't have examples of any entities that produce language-like output that doesn't have communicative intent. I suppose maybe parrots being the unusual exception there.

So whenever we encounter something that looks like language, we're ready to interpret it as if there's a communicative intent and some sort of intelligence behind it, even when it's a really dumb chatbot. And then with the current models, which are kind of, I would say, very smart chat bots, but still essentially chatbots, right? It's very easy for us to perceive perhaps more than there is there. But at the same time, I do think that there is there is some learning there to be had about what you can do with a lot of data and, you know, many simple nodes connected together, which in some sense is what the human brain does, right, in some very abstract sense. But I think we need to learn more about the specifics of how that works, because it's clearly different from how the machine learning models work. Well, I could connect with the viewpoint that I was articulating before, but it's not really my own position. My own position is that, you know, there's a space of possible minds that is vast, and human intelligence occupies one region of this vast space. And we're going to create, if we create anything that counts as intelligence at all, we're going to create something that occupies a different space, you know, in that possible, that space of possible minds. Maybe there will be

some overlap. Maybe we'll be adjacent, you know, areas in this abstract space, or maybe we'll be very far apart. But, you know, if the AI can do something useful, and, you know, it does so far, but I think there's a lot of unrealized possibility that people are assuming is going to be, you know, delivered upon very soon and I'm not confident that it will be. But yeah, to me it's not important. What's important not necessarily is that artificial intelligence replicates human intelligence. I think it's more important that it complements human intelligence. Well, that raises a big question about what we want artificial intelligence to do. I'm very interested in artificial intelligence as a model of human intelligence for scientific understanding. So for that purpose, I'm more interested in it,

telling us something about it. But of course, there are many engineering and practical applications for which that that's not really necessary. And there, I think we have a bunch bigger question about what we want artificial intelligence to sort of do for us. I mean, personally, I think that it's likely that artificial intelligence will drastically reshape our societies and economies over the next, I don't know, 50, 100 years. We don't really know how long a timeframe. It's already having significant effects, but we're still in very early days here.

And it's likely, I think that we will have a situation where just like, I mean, I'm not the first person to say this, just as in the past, we had a, a change with the role of humans in the economy through the Industrial Revolution, where artificial muscles effectively were able to replace human muscle power in many, many contexts, and humans then adopt the different roles in the economy. Likewise, we'll probably see many of the roles that humans have traditionally occupied be replaced with artificial intelligence, is probably for the better, and hopefully we'll find another place for humans in the economy. I guess some people are worried that AIs will replace us at everything. I think

that that's relatively unlikely, at least over the, you know, foreseeable future of decades. But I mean, that's, that's very much an open question. But yeah, to come back to your point, it's not, it's not obvious that those sort of AIs would have to think like humans to do many of the tasks that we would want them to do. Although I think that, at least for some tasks, thinking more like humans would be useful. And from a scientific point of view, yeah, well, then it becomes crucial if we want to understand how humans think. And if we want to upload our minds and live forever. Well, yeah, that's another aspect, right? So that relates to something else that I'm interested in, although not working on, which is whole brain emulation.

Now, this is very different from language models, which try to model high-level cognitive phenomena. A whole brain emulation would be a bottom-up simulation or emulation of the, well, the human brain at a neuron level or even a molecular level, depending on how ambitious you were. There are people who've talked about this, but we still don't currently have the technology to perform the simulation or a number. enough knowledge about what we actually would be trying to simulate. In theory, though, depending on which philosopher you believe, if we could perform a whole brain emulation to the sufficient level of resolution, then potentially that would basically just be a human person in silicate form. And so hypothetically, at that point,

there would be nothing to stop you from scanning your own brain and uploading yourself. And personally, I think that that will be possible, but that definitely requires a lot of technological advancements that we don't have yet. So Parker says, this talk makes me view chatbots as idiots, and so AI is not to be feared. I don't know that the second one follows from the first. Well, I mean, idiots can be dangerous, right? Or idiot technology. But it's also the people using the technology, which more is what I would be concerned about.

But the other thing is I wouldn't exactly say that the current language models are idiots. I suppose if you were talking to a person who thought like a chatbot, you'd think that they were an idiot, because they would keep forgetting about what you were talking about and they would say non-sequiters. Although there are people who kind of talk like that, so I suppose maybe it's not as unrealistic as who might think. But the other thing is they're still capable of very sophisticated linguistic behavior, right? And so, I mean, people probably heard about the Google researcher. I think it was a Google who recently was one of their bots became conscious. Blake Limon was the researcher at Lambda was the language model.

Yeah. So, I mean, they are capable of sophisticated behaviors. And, I mean, there are legitimate concerns there. One would be in terms of potentially stealing information from people online, like through confidence tricks essentially. That would be one application that I could readily foresee. Of course, there are good uses as well. So it's really a question of how the technology is deployed.

And to that, I would say that what we should fear is not the technology, but the use to which it's put in the institutional environment that it exists in, which, I mean, I think that's true for pretty much any technology of it. Well, I'd like to talk about economic issues, but you've been very grounded and responsible in your statements here about artificial intelligence for the last 40 minutes. So let me just throw open the gate and say, play wild-eyed futurist. And what are some colorful speculations that you're harboring about where we might be going in the near future with AI?

Well, machine learning models are becoming more and more capable of performing specific tasks that they're trained to do. And I think that at some point in the near future, probably, we will see some of these models be applied to perform. A lot of what I might call mundane white-collar work that people perform in kind of an office environment. A lot of this is things like, you know, updating spreadsheets and producing reports and stuff like this, right, which is sort of routine, but still requires a level of understanding that up to the moment is, well, actually, there is a whole space of people who have automated their jobs and they're asking on like Reddit whether they should tell their bosses and so forth. So actually, some of it could already be automated. But I'm saying beyond that, I think we will start to see probably of the next decade or two, this becoming happening, happening in.

increasingly at probably larger businesses, we'll start doing it first. And that will change the employment environment, right? Because skills that were previously marketable will cease to become marketable, they'll become increasingly more difficult. I think that what we'll see is that people will have to have more, well, I guess it's sort of the trends we've already been seeing. Basically, things that differentiate them from a machine, so ability to learn quickly synthesize information and think analytically in ways that it's still more difficult for machines to do. So I guess that's kind of already trends that we've been saying, but I think that we'll see an acceleration of those where people will have to increasingly differentiate themselves in these sort of cognitive

task from things that computers can do. I think that's going to bring a whole, will hopefully bring a whole lot of benefits. One of the things that a lot of people don't like dealing with is poor customer service. Hypothetically, it should become possible soon for, I mean, companies are already doing this, but we're not quite at the level yet where you can have a chat bot that can, you can just call up a helpline and it will basically deal with your, it will deal with your problem because very quickly you get into something that it doesn't know how to deal with. But in constrained environments like this, it should be possible to deal with a large percentage of these sort of issues in an automated way, at least fairly soon.

And I think these are the sort of applications that we'll see to come online first. Basically, when the environment is fairly constrained, and if you have an algorithm or a machine that's capable of producing an understanding natural language within its constrained environment and then dealing with a relatively constrained range of possible responses that will be able to see these machines deployed in those contexts. And that would be great, right, because we wouldn't have to deal with. I mean, I suppose there will still be an element of frustration, but at least you won't have to stay on hold for, you know, like two hours and have all the other annoyances of things like that.

So I think that there's a lot of boons available there, but it will create challenges in the labor market, which we've already kind of seen with outsourcing and globalization, but I think that these are only going to increase as many cognitive traditionally cognitive tasks then come under the realm of things that can be automated. But I do think that we'll continue to see an area of comparative advantage for human beings because algorithms are not cheap, computer power is not free, the intellectual property that these things will be under will still cost something. So there'll be a place for humans in the workforce for a long time to come. But I think that will be a challenge for a lot of people and we'll

probably also put pressure on our education system to continue to adapt to focusing on skills that are actually relevant for the workplace. Well, I'm tempted to, encourage you to push further out toward the fringe, but let's jump over to economics for a bit. And we can tie it back to artificial intelligence because they are certainly, you know, linked concepts. But I just listened to a series of podcasts you did, not just one episode, but multiple episodes in a row, talking about what causes some countries to enjoy the benefits of economic growth and others not. Yeah, yeah.

And as with many things, you know, the economic story that people are going to gravitate, is probably going to be a downstream effect of their political ideology. So, you know, if you're a post-colonial studies major, you're going to say that the rich countries are the ones that went to the, you know, the global South and raided it and took everything back and, you know, basically we're just living off the spoils of grand theft and everybody else is living in poverty because they had everything stolen from them 200 years ago. You know, I've stated that in a very prejudicial fashion. That's not exactly how it's stated on college campuses, but, you know, words to that effect. But what are some of the other theories that people have put forward as to why some countries enjoy rapid economic growth, a rapid growth, not just of, you know, accumulations of wealth at the top, but, you know, a widespread prosperity, you know, growing middle classes, rapid industrialization,

and other countries just languish. Yeah, well, I'm glad you to ask about this. this is one of my favorite series of podcast, even though it hasn't done as well as some of the other series. I think that many of my listeners are more interested in the physical sciences, which I think is somewhat a shame, because as I mentioned before, I see science as more unified. And this question about economic growth and development is central importance for our times because it shapes so many things that happens in the world and there's so much human misery that is still really avoidable if we could figure out how to generate sustained development in many countries. And so it's something that I've been interested in for a very long time and wanted to synthesize some of my thoughts and readings in a series of podcast episodes. And one of the interesting things about this question, at least in my experience, is unlike many other scientific questions or social questions, many people seem to think that they know the answer to this question. Whereas a lot of other things, people will acknowledge that they kind of don't know.

But at least a lot of people have sort of folk theories about the reasons for why some countries are rich and some are poor. And you just sort of mentioned one of them, which is about sort of colonialism and exploitation. What, most of the folk theories have found academic adherence and then like more developed forms discussed in the literature. But in addition, I think that there are other views as well. So one common view that you tend to find in maybe different areas of the political spectrum focuses on geography. And there are a number of recent slightly older books that have talked about the importance of geography for development. So the argument here is that nations that have experienced more development find themselves in areas on the coast so that there are more able to trade with more navigable rivers. in temperate regions rather than tropical regions, obviously without, you know, deserts and

without as many, like, impenetrable jungles or mountains and other things that promote ethnic fractionalization, which is known to cause issues. So that's sort of one argument that some people make. And there is some evidence to support that, but there are also, for any simple story about growth or development, there are countries that you can find or regions of the world that are exceptions to that, which is one of the things that makes this question so are interesting. So one of my favorite examples is Switzerland, right, which is basically the richest country on earth that doesn't have like huge oil supplies. And Switzerland is like the counter example to many of these stories because it's mountainous. I guess the rind flows through a little bit of it at the top

in the north, but for the most part, it doesn't have large navigable rivers through most of it. It's ethnically fractured because it has like four languages as used and it doesn't have significant natural resource deposits and it's landlocked. So it doesn't seem that it would have any of the, any of the core ingredients that you would typically think geographically. Another theory that's been discussed a lot is the importance of education. The, The argument there is that particularly primary and secondary education provides people with the literacy and numeracy skills and other basic knowledge that they need to function in an industrial environment and to seek the employment and find jobs and so on that then enables them to

build the economy and gradually develop a more knowledge-based economy. And there certainly is some evidence that supports that, but there are like other things, many counter examples as well. One of the things that this doesn't explain is the poor performance of the former Soviet bloc economies, which had very educated workforces, but certainly towards the later 20th century, performed quite poorly. And there are also other countries that have invested, developing countries that have invested in primary education and not necessarily seeing the returns that they would expect on that. So that's the education explanation. Another explanation is culture. This goes right back to the

19th century, the Protestant work ethic and ideas like that, where the cultural differences between nations explain why some people are more industrious and more focused on invention and innovation than others. A more recent example, of that is appeal to like Confucianism or Confucian values as an alleged source of success for China and South Korea and Japan, countries like that. This argument, although you still find it in the literature, is, as I say, taken somewhat less seriously these days. One of the reasons is because culture is so multifaceted that you can always find aspects of any culture that both promote that you would expect to promote economic development and also retard economic development. So it's a little bit hard to kind of construct any theory as to why one would be more important than others. So, um, but you still see this argument discussed sometimes. And the theory that I, in my series of podcasts, focus on the most, and I think is kind of the mainstream view, at least in economics, about the single most important factor favoring economic development is institutions.

So political and economic institutions. So institutions are sometimes defined as the rules of the game that shape how you or how firms and how individuals and groups function in a society. So these things, these include things like the legal system, protection of property rights, how the political process works and how the political ruling class or the ruling party or whatever is determined, you know, rules that shape the labor markets and the education system and all sorts of things like that, right? So this includes both formal rules and also informal norms and things like that. And there is a huge amount of literature

in economics and political science that focuses on how different institutions are formed and sustained over time and how they change and how some institutions promote, you know, investment and entrepreneurship and others make that more difficult. For example, a society in which corruption is widespread, it's often much more difficult to invest and much more difficult to ensure that your property rights will be protected and that you'll be able to maintain the control over your property or your innovations. And you end up having to make a lot of bribes to different people in the political apparatus to ensure that you can do anything.

And so that makes it harder for innovation to occur because you have to have the money to sort of bribe everyone. So that's an example of a type of institution that is not fair. to growth. So there's a lot of focus on that these days. One of the problems, though, is that we don't really have very good theory as to how you get good institutions. So it's a little bit, in some sense, a circular explanation because more developed countries have better institutions, but we're not entirely sure how you sort of go from the one to the other. But anyway, that's a bit of a whirlwind tour of some of the different theories that I discuss in more detail in that series of episodes.

Just to retread a little bit, you were talking about institutions as something that is important that would allow a country that starts off with low GDP to grow rapidly. But what are some of the other things? You mentioned geography. So the further from the equator, you know, a country is, the better off it is in terms of rapid growth that, you know, the equatorial countries tend to be poorer, you know, with some notable exceptions like Singapore. But, you know, for the most part, the hotter it is, the harder it is to, you know, get economic

growth going. You talked a little bit about education, and I was listening to this episode just earlier today, and I think there's one other category that I can't bring to mind that I don't remember you mentioning. Oh, democracy. Oh, yeah, that's right. That's another one that's discussed. Yeah. Talk about democracy. How important is democracy for economic growth? Well, in my opinion, not very, actually. At least not at the level of moving from very poor to less poor. I think it's probably important at the level of moving from middle income to higher income. And this is a question, I discussed this in my episodes. And it's a nuance that's important, but it's surprisingly often neglected in a lot of the discussion I see about this actually,

even people who should know better, that there are many levels of development that are possible for a country to have, right? In particular, we can distinguish between countries that have not yet industrialized. So in our world, that would be much of sub-Saharan Africa, still large parts of India and certain areas in Southeast Asia for the most part. And we might call those low-income countries. On the other hand, there are countries that have largely industrialized, a little non-industrialized pockets around, but still have not reached high-level income status and for which there's still large, you know, often large urban slums, large underemployment and many areas of the economy that are very backward and inefficient. But there is still large-scale urbanized.

and many people working in manufacturing and large primary level literacy. So these are middle-income countries, many countries in South America, for example, large parts of China and the Middle East would fit this sort of general scheme. And then there are high-income countries like in Europe and Japan and the US, right? So now the thing is that the factors that promote moving from being a low-income country to a middle-income country are not necessarily the same as those that promote moving from a middle-income country to a high-income country. And it's important to keep that distinction in mind because people often make comparisons.

across those sort of differences. So it certainly seems, so certain people have argued like Amati Sen, for example, that democracy is very important, although he focused on like famine relief in India, but the question of economic development is a little bit different. Some have argued that a democracy should be better able to promote sharing of resources and wealth and institutions that benefit the majority instead of a small minority. And there's certainly an argument to be made there, but empirically, it at least seems to me that there isn't much evidence for the claim that democracy promotes economic development in low-income countries. In fact, it's actually very difficult to find stable democracies. Let me rephrase that. It's very difficult to find low-income countries

that maintain a stable democracy over any significant period of time. The only real example that I familiar with is India, which has really been an exception there in maintaining a sort of a genuine, open democratic system, despite being very poor. They've begun to develop a little bit in the past few decades, but for much of its independence history, 50s, 60s and 70s, it was a very, very poor country. Very few other countries managed to do this. Most of the countries in Latin America and in sub-Saharan Africa and Southeast Asia that became independent quickly became authoritarian or single-party states or Marxist states, depending on the case, pretty much none of them maintained multi-party democracies. And so you don't really have, it just doesn't seem that low-income

countries are very good at maintaining democratic systems. And I guess, there's plausible reasons for this, right? That if you have mostly, if you're mostly undeveloped, if many people live in rural areas, they don't have access to much media from the wider country, if they're not literate. And if they are mostly concerned about day-to-day survival, probably they're going to have less interest and ability to engage in political dialogue. And it tends to be the case that existing elites then co-opt those systems and then use it for their own purposes. There's also a lack of democratic norms and institutions in many countries at independence because they weren't sort of prepared for that.

But so it's sort of hard to see much of an effect of countries that are democratic then becoming developed. Indeed, it seems to be the opposite. It seems to be what will happen is that a country will become developed and then it will become democratic. So this happened in Taiwan. It happened in South Korea would be two clear examples where they achieved relative level of prosperity and then democratized, I think in the 80s or 90s in both of those cases. I guess it's kind of happened in Eastern Europe as well, although obviously you have the cold. war issue there that complicates things a bit more. It's happening countries like Chile as well that democratized more recently. So yeah, it seems that in many cases the development, at least to a

certain level, happens, and then you see democratization. And indeed, just as there are very few underdeveloped countries that are democratic, there are very few highly developed countries that are non-democratic. Obviously, some are more democratic than others, right? We see democratic backsliding countries like Hungary and Turkey recently, for example. But to full-on authoritarian regimes, you really just have really have to look at the petroleum states of the Middle East, like Saudi Arabia, for example. And those are always kind of an exception, right? Because they're very rich, but they're not exactly developed in the same way because their economies are so highly focused on oil. And they have huge numbers of foreign laborers and other things. So if you put those to the side, basically all developed countries are democratic, at least to an extent. And basically all low-income countries are undemocratic. And then in the middle income, you have kind of a mixture of both.

So it's a bit difficult to tell a clear story about the relationship between democracy and growth, other than at least to my judgment, it seems that democracy follows from growth rather than causes it. And I actually think that one of the reasons for this is that because when people have their needs met to a certain extent and receive a certain level of education and ability to participate in the political sphere, it's at that point that they begin to demand more freedoms and more representation. That's what we saw, for example, happen in the former Soviet countries when their standards of living reached, you know, at least like modest levels. And then they began to compare themselves to the West and realize, you know, we don't have things that are good here and we don't have the same freedoms that they did here and began

to demand for more freedoms. Obviously, there's more to the story than that. But I do think that's a part of it. So, yeah, the story about democracy and growth is a complicated one. Well, equally complicated, I think, would be the relationship between petroleum or, you know, fossil fuel resources and growth. There's something called the resource curse. Some places that are seemingly blessed with, you know,

great stores of rare earth in minerals or oil or natural gas, they're nightmare places to live. And other places, you know, with that seem equally blessed like North America are, you know, models of, well, maybe not models, but are comparatively much more egalitarian and democratic with strong institutions. So what's, what's it worth? there? Why do some countries fall prey to the resource curse and others just benefit from their resources? I don't think anyone knows the answer to that. So most countries that have very large

deposits of concentrated high value materials like petroleum or diamonds would be another example. Most of those are authoritarian countries. And it seems, so there's an argument in the literature that the ability to extract the rents, essentially, which just means free money that you don't have like compete for. To extract the rents of those valuable minerals or natural resources is what enables the authoritarian regimes to fund, to fund the system that keeps him in power, basically. So they need to pay off the military and other key supporters, but they don't have to provide, at least they don't necessarily have to provide a broader society that's inclusive and brings more people into the political system. So that's an argument that's made. I mean, one problem is

that authoritarian systems also arise in resource poor environment. So I'm not sure that the resources necessarily predicts one or the other there. The other thing is that there are some exceptions, right? So one example would be Botswana, which is the big exception in sub-Saharan Africa. It's really the only sub-Saharan African country that has achieved sort of middle-income status. I guess South Africa would sort of be the other one, but there was a large settler colonial population there, so I kind of put that in a separate category. But in terms of countries without any significant settler colonial population, Botswana is really

the only one to do very well. They're not high income, but they're much wealthier than surrounding countries. This is despite the fact that they are landlocked, that most of the country is desert, that they have suffered from a huge HIV-AIDS epidemic. I mean, still do, but it's a little better than it used to be. That's devastated their working population. And they have huge diamond deposits, which, based on the resource course, you would expect to be, to lead to civil war and authoritarianism. That's the other thing that I should have mentioned that apart from leading to authoritarianism, readily available resource rents also tends to, the argument is promote conflict because people fight over who. controls the resources. But that hasn't happened in Botswana. Botswana has actually been quite

peaceful and relatively prosperous compared to its neighbours. I mean, it has had much more effective institutions than surrounding countries. But of course, the question is, well, why is that? Why have they been able to do better and manage those resource rents better? I don't know that there's a clear answer to that. And I think what we do need in this area is more focused, detailed work on looking at particular case studies and trying to fit those into broader theories. There's still a relative of that kind of work in my view. But I think overall it does seem that readily available natural resource rents are bad for many countries. The USA is a bit of a different case because although it does have like it has large or had slash has large fossil fuel reserves, its main,

the main reason that it was settled and became prosperous early on was its large arable land, not so much as natural resources. That actually came later. So its institutions and wealth was actually developed primarily in sort of 19th century on the basis of its, available land and then it began to industrialize in the second half of the century. And so it didn't enter the modern scene, so to speak, with those readily available resources present, which is different from cases like, say, Saudi Arabia, which basically became a country just, well, slightly before the large oil deposits were discovered and began to be exploited. And then there's cases like the Democratic Republic of Congo, which is really just a very dysfunctional country

in the center of Africa that has never been prosperous and has been badly exploited throughout its history and has seen almost continual civil war for decades, but has huge natural resource deposits of all sorts of things. And it's a real tragedy that things have gone so badly for that country. It has almost everything working against it except for the natural resources. It's very large, so it has many different ethnic groups. It's had a history of exploitation, so it's never really had any kind of government that works for its people.

And most of the territory is like tropical rainforest, so like navigation is difficult, although it does have the Congo River that runs through large parts of it. So I think the answer is that really you need to look at different countries and even regions within countries, kind of a case-by-case basis because there are so many different contingent things that happen there. At the same time, you do need a theory to be able to interpret what you're seeing there. So it's kind of, and that's why, as you mentioned earlier, so many people come to the table with views that are based on their political theories.

It's because you kind of need a framework, right? But I guess the question is trying to ensure that it's more of an objective framework that's based on the evidence and not just your own preconceptions. So, I mean, it's clear that in coming back to natural resources, it is clear that often they can be a bad thing for a country, but they can be a good thing if they're well utilized, like the wealth in Botswana as an example. And I think the answer to that is you need to develop good institutions that are able to put that wealth to good purposes. But how you build those is really an open question. And I don't think anyone has a good answer to that. I mean, if we did, we would have done it, right?

It's perhaps not entirely surprising. We know what good institutions look like, but we don't quite know how to develop them from a baseline where there aren't any. Parker in the chat points out that you said that, you know, the United States and Canada, they were not based on, you know, petroleum. That's not why they became a country, that they were, you know, well-developed agricultural economies before petroleum was even relevant to anybody's, you know, thinking. But Parker points out, even before we had the agriculture, there were beavers. Lots of wealth was created trapping beavers here in North America and sending the pelts to Europe. Yes, the fair economy. That's true.

That's true. All right. Well, hey, James, we've been on for a good long time, so we're going to have to wrap it up. But I very much enjoyed this conversation. I hope it is the first of many. Yeah, it was a pleasure. Thanks very much. And it was good to chat about a diversity of topics, which fits my podcast, I suppose.

It does indeed. All right. So we'll have links to your podcast in the show notes. Is there anything else that you would like the listeners, to be aware of that you're working on. Yeah, well, if people are interested, you could check out my podcast, as I already mentioned, the Science of Everything podcast. But I also have a blog, which is called The Godless Theist, which is mostly focused on philosophy topics, because I'm also interested in philosophy and religion. I also have a YouTube channel. That's just my name, James Fodor,

where mostly that's about philosophy, but if you're interested, you can check that out as well. All right. Thank you much. I've enjoyed our conversation. It's a pleasure. If you enjoyed this discussion, you may be interested in some of the other content on the Padverb podcast. You can check them out on iTunes and other podcast aggregators. It's probably easiest way to find them is just by Googling Padverb podcast. That's P-A-D-V-E-R-B podcast. Next episode, we will go back to regular scheduling. Thanks very much for listening.

I will talk to you next time.