Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas - 260 | Ricard Solé on the Space of Cognitions

Episode Date: January 1, 2024

Octopuses, artificial intelligence, and advanced alien civilizations: for many reasons, it's interesting to contemplate ways of thinking other than whatever it is we humans do. How should we think abo...ut the space of all possible cognitions? One aspect is simply the physics of the underlying substrate, the physical stuff that is actually doing the thinking. We are used to brains being solid -- squishy, perhaps, but consisting of units in an essentially fixed array. What about liquid brains, where the units can move around? Would an ant colony count? We talk with complexity theorist Ricard Solé about complexity, criticality, and cognition. Blog post with transcript: https://www.preposterousuniverse.com/podcast/2024/01/01/260-ricard-sole-on-the-space-of-cognitions/ Support Mindscape on Patreon. Ricard Solé received his Ph.D. in physics from the Polytechnic University of Catalonia. He is currently ICREA research professor at the Catalan Institute for research and Advanced Studies, currently working at the Universitat Pompeu Fabra, where he is head of the Complex Systems Lab. He is also an External Professor of the Santa Fe Institute, Fellow of the European centre for Living Technology, external faculty at the Center for Evolution and Cancer at UCSF, and a member of the Vienna Complex Systems Hub. He is the author of several technical books. Web site Google Scholar publications

Transcript
Discussion (0)
Starting point is 00:00:00 If you're a QuickBooks customer looking to grow your business without the growing pains, you need the Intuit ERP. Upgrade to Intuit Enterprise Suite in a matter of hours. It's the AI Native ERP from the makers of QuickBooks. Learn more at Intuit.com slash ERP. Hey, everyone, it's Cal Penn. I'm inviting you to join the best-sounding book club you've ever heard with my podcast, Earsay, the Audible and I-Heart Audio Book Club. Every episode, I nerd out with amazing guests and dive into the best new audiobooks available on Audible. It's the book club for your ears. Listen to Earsay, the Audible and IHeart
Starting point is 00:00:37 audiobook club on the IHeart Radio app or wherever you get your podcasts. Hello, everyone. Welcome to the Mindscape podcast. I'm your host, Sean Carroll. You all know, if you're a listener here, that I am on the complexity bandwagon. I think that complex systems are really interesting. I mean, maybe that's just kind of obvious. Maybe everyone thinks complex systems are interesting. The question is, can we make progress thinking about complexity in and of itself? In other words, there is a human being, here is an economy, here is the Milky Way galaxy, these are three complicated systems. Do they have anything in common? Does it make sense to study the idea of complexity as a field of study rather than just studying the individual examples of it separately?
Starting point is 00:01:29 So I think that it does make sense. And I think that it does make sense. that we don't have the answers yet. We don't have a fully fleshed out theory for how best to think about complexity. I did the recent podcast with David Crackauer, where I suggested that maybe we should think of complexity science as pre-paradigmatic. He didn't like that. He says he thinks that there's a paradigm out there, which is great. It's great that people disagree about this. I mean, maybe we actually agree on the substance and are just slightly disagreeing about the words. But one way you can make progress on thinking about complexity is to really narrow in on a particular kind of complex system and study it from all angles.
Starting point is 00:02:09 And if that's true, then what better kind of complex system to study than the brain? Or not just the human brain in its biological specificity, but the idea of an intelligent brain. Okay? So, of course, you can study the brain. You can be a neuroscientist, et cetera, et cetera. But you can also take a step back. You can abstract. You can say, okay, I think that a human brain.
Starting point is 00:02:32 is intelligent, it has thoughts, it does reasoning. We are also kind of talking about artificial forms of intelligence, right? So what are the general principles here? What are the general circumstances under which a system of any sort can be thought of as thinking, as doing cognition, as being intelligent? What is the space of all possible cognitions? And how do you get there? Right? How do you get there in biological evolution? How do you get there in design? What are the different phases? So, do you need it to be kind of a solid structure? You know, the brain and the human being is kind of squishy, but it's still mostly solid. The neurons are hooked up to each other in more or less a predictable way, whereas if you look at the flight of starlings, for example, right, a flock of birds
Starting point is 00:03:25 communicating with their nearest neighbors, can you think about that as a kind of collective intelligence? What about ant colonies or bee colonies? Is there some thinking going on? Is there some intelligence that is not from individual neurons hooked up in a rigid way, but rather from individual units that can move around and flow into each other in different ways? So you have solid brains, you have liquid brains, you have artificial brains, what's going on? What is the space of all possible ways of thinking? Well, if that's interesting to you, you've come to the right place. That's what we're talking about today with Ricard Soleil, who is the head of the complex systems lab at the Catalan Institute for Research and Advanced Studies in Barcelona, also external professor at the Santa
Starting point is 00:04:11 Faye Institute, and he's trained as a physicist, but like many people in this area, he has let his curiosity fly around, and he's ended up thinking about exactly this question. How complexity develops, what is the space of possible cognitions, what kind of architectures are there? for information to flow in a network that you and I would recognize as something intelligent, something doing some kind of cognition. That might help. This kind of thing might help not only in thinking about biology, but in designing intelligent agents, whether it's artificial intelligence in a silicon-based computer, or maybe synthetic biology when we're in there editing the genes to make new kinds of organisms that might qualify as being intelligence. So again,
Starting point is 00:05:00 very much along the lines that we support here at Mindscape, which is that the basic research into these grand concepts will, if you do it correctly, pay off down the road in specific ways of thinking about really down-to-earth systems. So Ricard is a great guy to talk to about these things. We go over a while, a bunch of things. It's a lot of fun in the conversation, so I hope you enjoy it. Let's go. Ricard Solet. Welcome to the Mindscape Podcast.
Starting point is 00:05:44 Thanks for having me here. So the space of cognitions is the phrase that has appeared in talks that I've heard you give and articles you've written. That's just a very exciting concept, the space of cognitions. Do we understand that very well? Is this a well-known thing? Are we just trying to develop the concept? Yeah, no, it's still an ongoing research. The ambition here was when you speak people, you know, you hear people speaking about, you know, cognition in very different kinds of biological systems. Oftentimes you feel like things should be much better defined. And something that we launched at the Santa Fe Institute in a workshop was the idea of trying to kind of map the cognition space by including things that go from what we call the solid brains, which means our brain, for example, with neurons located in specific positions and where all the fun happens in the interconnections.
Starting point is 00:06:48 Right. Whereas in nature, you have all colonies, the immune system, it's plenty of interesting stuff there that doesn't involve that kind of picture of fixed neurons in space. But instead, they move around and they process. information in different ways. So how do we put all this together? And that's kind of the ambition. And just so that the audience knows sort of how to triangulate where we are here, this sounds pretty darn interdisciplinary, right? Like actual brains, but also collective brains and ant colonies or the immune system and also AI kinds of things, right? It sounds like a lot of training is
Starting point is 00:07:29 involved in figuring this out. Yeah, of course. And in fact, this ambition of mapping the cognition space come from our interesting understanding whether or not there are general laws for complex systems, in particular general laws that define or constrain the possibilities that evolution can explore
Starting point is 00:07:52 in terms of language, cognition, sentience. And yeah, I mean, this is a very much interdisciplinary effort an artificial intelligence comes now as an interesting item, because of course you can ask now, maybe much better than two decades ago, whether or not
Starting point is 00:08:12 so-called artificial intelligence will be similar or not than ours. And a way, contributing to the question that I'm really, really fascinated about is what kind of possible things can evolution generate and whether or not
Starting point is 00:08:29 this big convergence. So even for artificial intelligence, maybe you'll find out in the end the same kind of design principles. We'll see. Well, that's good because I was just going to ask. You mentioned the possibility of general laws governing the behavior of complex systems. That's a pretty frequently mentioned ambition in the complex systems spaces that you and I move in. So what do you think?
Starting point is 00:08:52 Are we going to get general rules? Or is it more like we should settle for a bunch of specific rules applying to different contexts? I hope we do. I mean, of course, it's not a simple task at all. Even, as you know, even agreeing in what exactly complexity means. Sometimes it's a bit controversial. For me, the definition comes from emergence, essentially. This idea that like in a termite nest, termites construct these amazing structures,
Starting point is 00:09:23 which are thousand times larger than the individuals. Whereas the individuals are blind, they communicate in a very simple way. So you can spend your whole life looking at the individuals. You'll never figure out how the collective creates the hyperstructure. And besides that, I mean, there's a lot of ideas that comes from very different areas, and that's kind of the part of the trip we need to figure up from theoretical or statistical physics and computational theory. And we have to blend concepts from all of them. But personally, I think, for example, sometimes people discusses the idea of whether or not life in another planet or alternative biosphere will be different from the one we know.
Starting point is 00:10:09 And I pretty convinced that it might seem different, but the basic logic of life is probably universal and unique. Okay. Well, that's a very good open question, and hopefully we will get some data on it within our lifetimes. That would be very exciting. So speaking of life then, I mean, let's back up. We're talking about the space of all cognitions. That gets us excited, a little bit of foreshadowing, but let's just get to cognition in the first place.
Starting point is 00:10:35 Can you say, I don't know how you yourself think about the evolution of cognition. Is it just one in a series of major transitions that happened in evolution, or is it something special? Yeah, that's a very good question. Well, I think that if you think in terms of, again, general loss of complexity. One of the things that I think, and that's where cognition is so important, that is probably a part of the explanation of why the biosphere is complex and not, as somebody said, what is not just field of microbes, right, or replicate there's very simple
Starting point is 00:11:11 where is this whole complexity coming from? And the answer to that is that there's a big payoff in predicting the future and being able to gather information from the environment and respond to that in adaptive ways. And that propels brains. That's the engine of cognition in evolution. And that can come from this first simple cells, where we still need to figure out when those cells were able to use information and when those cells were able to do computations, because this is something that is at the, I will say, in the growth road between biology and
Starting point is 00:11:52 physics. And then I will agree of what you mentioned that across evolution, I think there are several cognitive transitions that have been happy life. So there's, I always think of it in terms of a pay, what is the word tradeoff kind of thing, right? Where as you say, you want to be able to predict the future. The world is a scary, unpredictable place. It's complex all by itself, even without organisms in it. And so we might want to process information and make predictions, but that costs energy, it makes us vulnerable in different ways. But it evolved, nevertheless. What do we know about that evolution?
Starting point is 00:12:31 How does one little neuron help us? Well, if you look at the basic units, clearly the invention of neurons is one of the big transitions. You know, every single cell we know, or the systems that are far from equilibrium, that they maintain themselves because they create these gradients between outside and inside.
Starting point is 00:12:58 And that's kind of the first step that was something that cannot only create this difference of ions from the internal and external medium, but use those to propagate information. And I always tell my students, actually it's beautiful to see neurons as this kind of specialized cell type. You see cells from the liver or the kidney,
Starting point is 00:13:24 they had this special functionalities, which have to do with metabolic things. But neurons, when you look at neurons, you see something that wants to communicate. That's why the jap is as it is. It wants information. And that was a big revolution in the evolution. So going back then,
Starting point is 00:13:44 is there any sense in which a single cell? thinks about the world, a single-cell organism, I should say? Well, it's a bit of controversy here because I personally think that it's too often, in the last decades, two decades I will say, it's too often misuse a term like sentience or understanding from single-cell organisms. I will say that we have a good deal of understanding. of what they do. And they do use something that we believe was kind of fundamental in some of the revolutions, Towers, Scott Mishia, which is associative learning, this capacity of learning
Starting point is 00:14:30 and connecting different kind of external items. And they can do that, and plants can do that. But I personally think it's not much more than that. But it's not the kind of the kind of memory storage that is so spectacular. in other aliens. But even that sounds kind of impressive to me. I mean, what does it mean to say that a single cell, a bacteria, I guess, does associative learning? Uh-huh.
Starting point is 00:15:03 It is, it is. So the thing is that they can gather information and you can actually engineer that in the levels. We could put complexity there so that they can gather information in a natural way about some kind of signal. but under an environment where there are some external stress that appears correlated with that signal, they eventually are able to recognize that stress signal, right? Decorrelated from the natural original signal that they had to be situated. I see, okay.
Starting point is 00:15:35 So it's kind of the Padlov Docs, the version and single cell version. I see. So there's something bad, and they recognize there's a correlation between this bad thing and this other signal, and then they start responding to the signal whether or not the actual original bad thing is there and that's learning. Okay, I get it. That actually makes sense.
Starting point is 00:15:54 How do they do that? Like, where in the cell do they store that information? I get the feeling from talking to you and other people that cells are more complicated than I give them credit for. Yeah, they are very complicated. And as you probably know, I mean, we actually talk about the genome of cells that are very small, kind of the minimal cell.
Starting point is 00:16:16 and we still ignore a lot of what many genus there do. But the thing is that they have these signaling networks that gather information from the membrane. You can imagine as a kind of cables going from the membrane into the genome or the nuclei, depending on kind of something is that. And in a way, it reminds us kind of a neural network, except that this neural network is just an analogy. I mean, whatever it is there is being fixed by evolution.
Starting point is 00:16:49 And then they can store information, for example, in switches. And within cell, we have switches that allow to store bits of information. My best skin ever at 45? Give me a theme song and a best skincare award because it feels like this, right there. That's farmhouse fresh skin, all right? I'm blowing. And everyone asks how.
Starting point is 00:17:17 The best skincare is Farmhouse Fresh, and the award is you, your best you. Visit Farmhouse Fresh skincare.com and use code radio for a free starter routine with any purchase. When Toyota builds an electric vehicle, we don't start with a blank slate. We start with everything we know. The BZ brings Toyota's proven engineering to electric, with impressive range, intuitive technology, and Toyota reliability, BZ reflects decades of experience, reimagined for what's next. The BZ isn't just electric.
Starting point is 00:17:53 It's Toyota Electric. We make it easy. Toyota, let's go places. What kind of literal switches are we talking about? Like a chemical that can be one way or the other? I'm not like, what kind of, what's a switch? The usual thing is genetic switches. Okay.
Starting point is 00:18:10 So, and actually the logic of that, we know it very well, usually switches that involve two genes that regulate negative each other. So I try to inhibit you, you try to inhibit me, to repress. And that allows to store memory, right? It's pretty much in a way what happens in electronics. Okay. Is it actually editing the DNA or the RNA or just expressing it differently? Well, in gene regulation, it's expression, right?
Starting point is 00:18:40 Okay. Whether or not a given protein appears or not. So, I mean, that way is kind of binary. Okay, good. All right, good. I've learned something already if our connection dies now. It would be worthwhile podcast. But I want more.
Starting point is 00:18:54 So do we know when along the evolutionary progress the first neuron came to be? Okay. It's ongoing discussion because it's very recent research by a number of groups. the first neurons have to appear in a context that has to do with multicellularity. Because neurons by definition, the least is that, and that's the origin of that. So at the beginning were cells that were able to act as receptors, and the cells themselves were able to detect something and secret some signal. But this still exists.
Starting point is 00:19:38 There's a lot of cells that do that. But you don't send signals in principle to anyone else. The beginnings is that receptors are able to kind of gather information in simple ways, but that information within the organism is not kind of propagating all around. Got it. For that, you need for the revolution. Okay, good. So the first, I always like it when something complicated can be broken down into steps
Starting point is 00:20:03 where you can see why each step would make sense. So first, we receive a signal from either the outside world or maybe. even inside the organism itself? It could be, yes. Yeah. And then we just do that something with it ourselves. But then the next step to being a neuron is talking to other neurons, I suppose. Exactly, exactly.
Starting point is 00:20:24 In a way, and it makes sense that this happens eventually. I mean, before the Cambrian explosion that happened 50, 550 years, I have to remember, a million years ago. Before that, we had a biosphere essentially with very simple organisms that performed functionalities like filtering water, pretty boring stuff. But to evolve the first predators, which in a way was a big revolution that changed everything, you need to have sensors. And the sensors have to integrate the information.
Starting point is 00:21:02 And that's probably the key, and I think it is a pretty reasonable idea, that if you have to move in an environment, not just being settling out, but you have to move in an environment that is uncertain, you need to integrate information. And the fact you need to integrate and in a way predict, it's probably the engine that, you know, make brains happen. And the idea, is this before or after the idea of predation? I've been told that, you know, once animals or once organisms start eating other organisms, the whole bunch of new capacities need to be developed. Yeah, absolutely. We're talking about metazone, so multicellular systems.
Starting point is 00:21:47 Because, I mean, predation is also in the realm of bacteria, the other predator bacteria also. But for animals, yeah, predation came with ice and the nervous system. So once this started to be in place, you had a whole biosphere of poor animals. It's just sitting there unable to escape. And that promoted a huge armed race, right? Of developing defenses against predators, et cetera, and that changed everything. Yeah, okay, good. And the arms race centers, I'm going to boldly conjecture this, and you can correct me,
Starting point is 00:22:29 around information in some sense. I'm still trying to, for my own sake, figure out how the process by which organisms got better at using information. I mean, in some sense, a bacterium uses a little bit of information, right? A gradient is more nutrient in one direction
Starting point is 00:22:48 than another direction, but it's not really thinking about the information in the way we usually think about it. Yes, yes, right. I will say that there are two major events here that have to be considered. One is, as I was saying before, that movement was crucial
Starting point is 00:23:07 because without movement, you don't have predators, really, and that required a nervous system. And the other thing was the development of sensors. And for doing that and integrating information, you need another revolution, which is interneurons. Right? Okay. So the elements that they are not just detecting signals or executing tasks.
Starting point is 00:23:35 They are in the middle. They are connecting. And once you have that, you have this beautiful thing, which is information processing. So you can, from there, you can actually jump into the real big complexity. Right. And precisely because some systems like, I don't know, plants, plants don't have neurons, but they don't have also anything equivalent to interneurons. Yeah, okay.
Starting point is 00:23:59 Information processing elements, right? And because of that, not having that is a huge limitation in many ways. Got it. So I think, again, that's a concept that I never really appreciated the importance of. So it's relatively straightforward to imagine the usefulness of neurons that sense things. Likewise, neurons that send out instructions to the rest of the organism. But then there's a revolution when you invent just neurons that only talk to other neurons and can really, therefore, process information.
Starting point is 00:24:32 That's their job. Yes, exactly. When did that happen? Can we pinpoint that? Well, probably, I mean, before the Cambrian, we for sure we had organisms that have nets of neurons, not brains, not centralized control. but nets of neurons like Hydra nowadays or others very simple organisms or like a jellyfish. So you do have a network and you do have interneurons,
Starting point is 00:25:09 but that is typically connected with things that have more to do with locomotion and not exactly complex information processing. But that means that it was the basis of the current explosion also. Okay. And I guess that helps explain why the word network keeps appearing in this kind of discussion, right? I mean, you have cells. They can do some things, but hooking up those cells in an array in a network is a crucial step in truly processing complex information. Exactly, because then, then again, you can also have emergent behavior. You can also store memories in complex ways. And again, processing means that you have are still. started to have access to a space in terms of dimensions, a big space of possibilities, whereas you have only sensing and reacting, you're just limited to respond to the environment in a very predictable and simple way.
Starting point is 00:26:10 And this is jumping ahead a little bit, but despite the fact that there's a lot of excitement about neural networks and AI and things like that, I take it that a typical computer architecture, like the laptop that I'm using to talk to you with doesn't have this kind of network structure, right? There's not subunits that you would recognize as neurons. It's more homogeneous. Like there's a memory, there's a CPU, and they're doing different things.
Starting point is 00:26:37 Yes, the logic architecture is totally different. This is the for Norman architecture. It's common to our computers and it has to do with something that is easy to understand that in a way, typically information is being processed it in a sequential way. And in a way that is extremely efficient, with the hardware we have, but has very little to do.
Starting point is 00:26:59 But that said, I would like to mention that one interesting thing that happened when people started to build these computers with very dense arrays of microprocessors. that interestingly, one thing that was found out is that the web of connections, the network that connected in a very efficient way to reduce costs and signal processing turns out to have statistical properties that are pretty much identical to what we observe in parts of the brain cortex. Okay. And it's again, as we were saying before, if you look for a kind of universal loss,
Starting point is 00:27:42 It's interesting to see that the engineers who didn't know anything about the brain cortex ended up in a scaling law, the Wrens rule with the same kind of behavior than parts of the brain cortex. Again, suggesting that maybe there are really universal laws. And you mentioned the von Neumann architecture. Is that distinct from how a neural network works? It is. It is totally because, you know, in a neural network you have, on the one hand, it's formed by, elements that are, in the case of living systems, are polar systems, our neurons and have a polarity that send signals from one part to another in one direction, and are organized usually
Starting point is 00:28:28 in multi-layers. And information processes is highly parallel, something that happens in the rate, but not in a computer, even in parallel computers. Right. Okay, whereas, I mean, maybe explain to our audience what the von Neumann architecture is in contrast with that kind of network point of view? Well, Phenema architecture is grounded in the fact that you have kind of basic modules, a central processing unit to process information and memory where you actually put the data ready for being processed there. And essentially it's an architecture that we identify very easily in our computers,
Starting point is 00:29:11 but as I was same before, in a way, it's inspired in the idea that you have to deal with software because that was actually the revolution that happened in von Neumann's time. Fornman was the one who actually foresee that. And the way of doing that for a system of this binary and using the kind of architectures we use, the simplest, nicest, and most powerful way
Starting point is 00:29:35 is using that kind of separation within processes. Okay, but, Nevertheless, so that sounds like a sensible thing for human beings to design when you first start designing computers, but you're hinting that, you know, as we're pushing our capacities more and learning new ways, we're kind of converging back on a more biological networked vision. Yes. In fact, we wrote a paper recently that we entitled Evolution of Brains and Computers. the road not taken. And in that paper that I wrote with one of my former studios, we saw on it, we argue that when you look carefully at the things that artificial neural networks,
Starting point is 00:30:25 and that includes the most common things they're using nowadays. When you look at the way they work and the potential that they have and what's really being deliberate, we defend the idea that probably in order to getting to the real general artificial intelligence, something that really matches what we do. You probably need to go through some of the paths that our brains have followed in evolution. And in particular, there are several things
Starting point is 00:30:55 that I think are extremely different and not yet there in the machines that are kind of the singularities of our brain. One is language, complex language. As much as you see that they can use language which is not the same kind of thing. The other is something that fascinates me and is that it's time.
Starting point is 00:31:17 Somebody said we are mental and travelers, right? That we on the one hand use memory, and the same architecture we use for memory allows to do something absolutely amazing, which is thinking in not one feature, but many possible features. That is a revolution, a revolution, really, in an evolution of humans.
Starting point is 00:31:37 And then this apparently disconnected, but very important thing, which is this capacity of understanding the mind of the others, right? Of understanding what the other is thinking, so to speak. Because when you put these three things in connection, something really singular happens. Yeah, absolutely. Nothing like that is in the artificial intelligence that we have. Okay, good. I like to maybe expand on that a little bit because I'm a big believer in the importance of the mental time travel stuff. We had Adam Bully.
Starting point is 00:32:08 I don't know if you know him, but he was a guest talking about how that helps distinguish human ways of thinking from other people. But you said the AI way of thinking is not the same kind of thing. So you just identified some features that are true for human cognition. In what way is AI not doing that? Because we all know, I'm on your side here, but anyone who has interacted with chat GPT knows that it sounds human. So how can it be sounding human if it's doing something so different? different. Well, I guess it depends who you ask. I like this idea. I mean, I'm impressed by chat GPT. I don't want to do this mischief here because I can't. Yeah. But it's interesting to see that
Starting point is 00:32:52 this system who has no past. So it's no childhood or learning or anything connected with other, as in humans, other people, right? Which the cultural part is, is enormously important, but they have been trained in making, so to speak, a cultural compression process, that in the way of doing something apparently so trivial, which is predict the next word, because that comes from this idea, how you predict the next word. But it turns out that what happens is that, and I think it's important to try to understand it, that in the process of optimizing this prediction, these systems
Starting point is 00:33:40 seems to have been generating something that is kind of reasoning, kind of something that mimics reason. Yeah. And I say mimics, because, of course, there's no understanding. But it's interesting to see that for us, the humans,
Starting point is 00:33:57 we are kind of looking there. And I always think in the origins of artificial intelligence. I mean, I was thinking, how do we see? the machine operating and how do we interpret that. And I was thinking when I was a student and there was this program, Eliza,
Starting point is 00:34:15 which was kind of a very simple program, right? Yeah. But even for us, I remember my colleagues that we programmed that. And even for us, knowing that, there was no intelligence, there was nothing there, right? But it's kind of something that calls to your brain
Starting point is 00:34:31 and kind of have a feedback with that machine. And chat GPT, of course, has amplified this in ways that we wouldn't expect. But again, in terms of there's no time there, I mean, clearly. And actually, you can actually test a little chat GPT with questions and things about time and see that there are some troubles there. But as more as we move more and more into big versions, I mean, clearly the lands get broke. Yeah, it gets better.
Starting point is 00:35:04 We'll see. will see. Okay. Let's get back to the biology a little bit. I'm not quite done with that before we move on. Because you referred to the kinds of structures that are in ordinary biological brains. I presume that what you're gesturing toward is the claim that the brain is a scale-free system, that it's on the edge of criticality, or it is critical, it's the edge of chaos. If I'm right about that supposition, explain what all that means and why it matters. When Toyota builds an electric vehicle, we don't start with a blank slate. We start with everything we know.
Starting point is 00:35:44 The BZ brings Toyota's proven engineering to electric. With impressive range, intuitive technology, and Toyota reliability, BZ reflects decades of experience, reimagined for what's next. The BZ isn't just electric. It's Toyota Electric. We make it easy. Toyota, let's go places. Okay, there's one clear thing in the brain in terms of dynamics, which is this critical state meaning that pathological brain states looking at the dynamics, right?
Starting point is 00:36:21 Well, either you can actually measure from EEGs or from any kind of non-invasive method, right? You have time series of changes in an epileptic state or in some kind of. of pathological stage, you see the brain state much more organized, much more regular. Right? Not good. Not good at all, of course. Whereas you are in, for example, coma states, lock-in states, you see that there's low activity, much more random.
Starting point is 00:36:54 So you have kind of a disorder state. We don't want that, of course. And the nice thing of the research has been showing that the healthy brain seems to operate on a critical state right on the boundary of a phase transition. It's ongoing discussion of what kind of exact transition is going on there, but clearly it happens to take advantage of the regularities that you have because you have internal oscillations, but this amazing capacity that you have the critical point of reacting quickly into any kind of stimuli. That's it. This is a dynamic.
Starting point is 00:37:35 state. How is this connecting with actual cognition? Right? Okay. Because, for example, language also exhibits some features of scale-free behavior. Other attributes that you can find out seem also, but how are they connected, right? It is something because in principle you could think in an intelligence that doesn't use criticality. So,
Starting point is 00:38:07 we need to know. You're kind of in sinuating or implying that they are connected, but maybe you're saying we don't know yet. That's a conjecture. Yes, yes. We don't have yet a good theory to kind of connect the computational tasks, as you will describe a functional cognition and the dynamics. Okay.
Starting point is 00:38:32 I don't have that yet. Good. But I mean, maybe one thing just to get clear for the listeners, because you and I will use a word like phase transition. But I bet that some people think that that's a process unfolding in time, right? Like water boiling or ice melting. And that's not what you're saying is happening in the brain, I presume. No, no, no. I mean, we do observe phase transitions, in particular in some kind of experiments that you prepare for people, for example, looking at the,
Starting point is 00:39:03 an object that has two different interpretations. You can see the branch flipping between different states. But it's more like at right of a phase transition where you have this critical state. You kind of live there. You stay there. And again, that's an interesting thing that seems common to other sisters, right? Like RNA viruses that live on the age of catastrophe, right, in the order on desert. So it seems to life kind of lives it.
Starting point is 00:39:29 Yeah. And I'm never completely clear on whether that should be. obvious or surprising in the sense that, you know, there's, there are more ways for the brain to be either completely disordered or completely ordered. Like, there seems to be some need for regulation to keep it right at this critical point where there's something going on at very different light scales and timescales. Right. I mean, it's, it's, as I said, since we're still lacking a theory, as, for example, we lack a theory, a neural theory of language, right, which is something that we really need to figure up at a toolbook.
Starting point is 00:40:08 I like the idea that actually Langton and others kind of introduced many years ago, that computation in complex systems, in competition in biology in particular, needs to occur somewhere where you can take advantage of the order that you need to store information and have, you know, regularities that are predictable. but on the other hand, end of the open-ended and we're able to actually manipulate information.
Starting point is 00:40:38 And probably on the boundary is with the order and disorder is what this happens. And for a physicist, the natural language is thinking that you are on the border of this phase transition between border and disorder.
Starting point is 00:40:52 So let's start then applying this to trying to figure out the space of cognition. So these are examples of brain and neural systems, and we're kind of familiar with that. You want to say that there's a whole other world out there, that these are solid brains, and what about the liquid brains? Yeah, yeah, yeah.
Starting point is 00:41:13 The liquid brains is a whole story. We think, for example, we tend to think in humans that we have been, unfortunately for our planet, we have been very, very successful in one of the reasons for that is that we are what we call in ecology ecosystem engineers. we are being able to manage to transform the planet, changing the flows of energy and matter to massive scales. It's interesting that we have a competitor here, which is social insects.
Starting point is 00:41:42 Termites in ants, as Edward Wilson said, if humans were not here, that will be the planet of the ants. Because we don't realize that. They are another kind of intelligence that has been able to do also ecological engineering, massive scales. Interesting that, and I want to point out, when people think about intelligence in the universe, there may be planets where intelligence have emerged, but it's liquid.
Starting point is 00:42:11 And liquid intelligence cannot be as complex as the solid one. So there's not going to be signals being sent from those planets anywhere. But that doesn't mean that it's not intelligent, of course. And that can be extremely successful in transforming the planet. So in that respect, one of the things we've been investigating precisely and trying to do a good theory of that is what is the power of liquid brains, brains formed by individuals that move around, which in a way, as Dan Bennett said, is brains of brains, right? Because every unit only fact has a literary group. So by liquid, because I think that people are going to be imagining a glass of water or a cup of coffee and you're imagining in your mind a colony of ants or of termites.
Starting point is 00:43:06 They're liquid in the sense that the individual pieces move around and interact differently with each other unlike the neurons in your brain. Exactly. And so there's no individual identity between pairs of ants. there's not such a thing as a connection between two given ants that in a way is stable. It's not such a thing. And that totally changes the landscape of possibilities. But the same happens with the immune system. The immune system is a fluid neural network and it's able to learn, it's able to store memory,
Starting point is 00:43:40 of course, with a very well-defined functionality dealing with pathogens. But it is a network. And if you make a model of the immune network, it's not much different from a standard neural network. Okay. I don't know. Yeah, it is. I can tell you. And so it's interesting to see that on different scales, you actually can find out, for example, we feed an organism, another neural network that is, in this case, liquid.
Starting point is 00:44:11 And there's also the question, for example, with some particularly interesting things like the microbiome. right, that this huge ecosystem that we carry out inside, and that in a way makes the claim that we are a single species, end of nonsense, because we are carrying out a whole ecology. And we know the microbiome, which is just, you would say just bacteria, but they communicate with the immune system and indirectly with the brain. So, you see, the separation between the sonnet and the liquid is not so simple. because it might be that they kind of interconnect.
Starting point is 00:44:51 Okay, but you just said provocatively that the liquid brains are not going to be as intelligent as the solid ones. And I might have thought that the sort of extra flexibility of having the ants move around and talk to each other gives the capacity of potentially more intelligence than the fixed hardwiring in our brains. It could be great. But think about what the ant colony or the termicolony does. In evolution, it's an emerging of this superorganism that in a way warranties to seek about the same idea of how do you reduce the uncertainty of the environment? How do you predict? One way of doing that is creating a nest, having an internal environment that is stable.
Starting point is 00:45:40 but the ultimate goal here is to reproduce the whole story. That's why the life cycle of a nest of an uncolonial, if you want, is so similar to the one of a multicellular organism. Okay. And development from not a single cell, but from a queen. And then this grows. You need to monitor your environment, or until you have resources.
Starting point is 00:46:08 But all the cognition, in a way, it's placed in the colony reproduction and being able to actually, you know, be able to reduce asserted. This, our brains does it in a way. They do it in a lot of way. But our brain has this amazing potential for memory. And that's just because neurons have identity. Yeah, okay.
Starting point is 00:46:33 To a pair of neurons have a special identity. If you destroy that, we have kind of a proof of that. The potential for storing, for example, memories is extremely reduced. I see. All right. So then in that case, let me just ask the skeptical question here. Are we even playing fair by talking about ant colonies or termite colonies as brains? I mean, do they live up to the implications of being compared with the human brain?
Starting point is 00:47:05 Well, of course. I mean, one of the reasons that we came about with. this term when we made, we write this working group at the Santa Fe Institute. And the idea was, how do we map the space of clinician? How do we label things that way that is meaningful and provides kind of basic categorization of what is there? I can understand your question because, you know, the brain is a big claim. But on the other hand, I mean, still, you have a system that can store information, process information,
Starting point is 00:47:40 because there is information processing. And in a way, it will be kind of unfair, not allow them to do that. I must say that if I have to tell you the truth, this comes from when I was an undergraduate student and I was reading half-tatter's book, Get the Leisure book. And I was, I mean, I was in love with the idea
Starting point is 00:48:02 that, you know, an Ann Cologne in a brain have kind of very related things. So what could they do? Yeah, what can you do? Do ant colonies talk with other ant colonies? No, they
Starting point is 00:48:18 fight with other ant colonies. Okay, okay. They don't gang together to end colony societies as far as we know. No, no, no. They don't do friends that way. Okay. In other words, instead,
Starting point is 00:48:34 they can expand some Some species of ants can expand over, you know, vast areas going through entire countries, right? Yeah. In supercolons. Okay. All right. Well, good to know. So that sounds like an axis or, you know, one coordinate on the space of cognitions, liquid versus solid, right?
Starting point is 00:48:57 So I guess one question is, are there gaseous and other forms in that particular dimension? and then secondly, are there other dimensions? Yeah, as you know in biology, in some way, as a different from physics, we find exceptions. And there are organisms in our list that in a way look like outlayers. One of them is fissarum. Fisarum is this kind of mold that is a single cell, but you can see it with a naked eye
Starting point is 00:49:34 you can be as big as this table this yellow it's kind of an extraterrestrial thing yeah it's a little creepy yeah yeah totally creepy and it's a single cell in the sense that there's there's a single mass
Starting point is 00:49:50 it's a lot of nuclei inside the cell and in nature they can they can manage to it's always moving around and spreading it's again between liquid and solid because it's clearly they maintain a lot of structure
Starting point is 00:50:09 but this fluctuating and changing all the time as they organize. If you look close and make a picture, seems like a neural network because they have all these very complex venation patterns as they move around and they can detect sources of food and make decisions about which one is the richest
Starting point is 00:50:27 and go in a goal. When I say glow, changing morphology to exploit that source the most efficient way. And somebody thought, okay, let's use that. Because what if I put two food sources, like they like flakes, so it's easy to maintain. And I put them in the entrance and the exit of a maze. Oh, yeah. And I put fissarol there because I can cut into pieces, et cetera.
Starting point is 00:50:55 And in the end, fissarro, fluctuates, changes, changes, changes. And then you have a single tube that goes. the optimal path from entrance to exit. So there are two things to say. One is the beautiful thing, and so why is this so different, is that the computation is the shape. So really morphology is what says, I made this optimization, this is the solution.
Starting point is 00:51:19 So it's nothing similar to that anywhere. The other thing, because sometimes this is say like, well, look, Physaron solves mathematical problems. Well, yes, but it's the human. who define the boundary conditions. And that's a big difference, right? I put the labrille here, and I put Fisadu there, and Fisadon exploits this special capacity that has, in a way,
Starting point is 00:51:44 is least action. This is what is happening here. It's kind of finding out the shortest path. And you can use that, but we have to be aware that it can solve problems if we do prepare the problems properly. It's not like it's smart and go, in the forest doing calculations. My best skin ever at 45?
Starting point is 00:52:07 Give me a theme song and a best skin care award because it feels like this, right there. That's farmhouse fresh skin, all right? I'm blowing, and everyone asks how. The best skincare is farmhouse fresh, and the award is you, your best you. Visit farmhousefresh skincare.com and use code radio.
Starting point is 00:52:31 for a free starter routine with any purchase. When Toyota builds an electric vehicle, we don't start with a blank slate. We start with everything we know. The BZ brings Toyota's proven engineering to electric. With impressive range, intuitive technology, and Toyota reliability, BZ reflects decades of experience, reimagined for what's next. The BZ isn't just electric.
Starting point is 00:52:57 It's Toyota Electric. We make it easy. Toyota, let's go places. Right. And you're using the correct name for this beastie, but is this what we call a slime mold informally? Isn't it the same class? Same class of slime mold. Because I know these stories of slime mold solving mazes, and it is a little creepy,
Starting point is 00:53:20 but it's a good example of a different kind of cognition. So, okay, so there's, there are examples that span the space from liquid to solid and in between. So if we're mapping out the multi-dimensional space of cognitions, what other things should we be thinking about other than the liquidity, solidity transition? Well, plants, plants are of course extremely important elements here because plants have evolved in a way that if you're thinking what I said before, that movement is something that has been the engine of building brains and evolving brains. Plans do not move. Plants have this special status that they can gather energy just from the sun. You don't have to move. They have an
Starting point is 00:54:08 enormous morphological plasticity, meaning that if I ask you how many organs you have, you just can count them because it's totally regular. How many organs has a plant? And since every single leaf is an organ, but they can appear and disappear, it's totally flexible. It depends on what you need. And that makes it always different from anything. They don't have neurons, not anything. It's been claims on that, not anything similar. So my position about that is that plans are absolutely amazing.
Starting point is 00:54:46 They really have terraformed their planet and they are spectacular in many ways. I don't think they like Mozart. I think it's a different story. Okay, very good. But what about like different organizational architectures? You've talked about the critical brain. Are there, is that just the single right thing to do if you want to be intelligent? Or are there other ways of achieving this that might, that biological, whether real or hypothetical biological systems might contemplate doing?
Starting point is 00:55:21 Oh, okay. Well, I wanted to mention that also in Ant Colonies, at least in some species we observe also, the critical state, see fluctuations of activity that match very well that. In the flocks of birds, the ways they change in shape, it's been very well characterized. They leave in the critical state. They really move around because they are a criticality. So in a way that you see changes that are very predictable, but they're absolutely ready for any kind of external signal to react immediately.
Starting point is 00:55:56 What other kinds of things you find out? I have made a conjecture that, and I want in it, one of the things I'm doing as part of my research is trying to put this in a form of way. Right. That anything that is evolvable into cognition will be characterized by two things. One is that, as I mentioned before, in fact, that you have elements that are threshold elements so that when you send the signal, you're waiting. how strong it is and you react all or none,
Starting point is 00:56:30 depending on how big is the signal, as nearest, for example, and you will have a multi-layer structure. And again, one of the reasons that I don't think it's a kind of a surprise that the engineers building these artificial neural networks keep using what you have been finding out in the brain cortex,
Starting point is 00:56:55 example, the multilayered structure of the visual cortex, neurons that are flesial elements. Can you escape from that, really? Because I don't think you can't. And another thing that helps to actually make a good argument is that, you know, this whole area of artificial life where in principle, in silico, you got a bold thing that could be absolutely different from anything you find out in nature. Why do we don't find any kind of cognitive system that in a way deviates for what we see? I bet that this is because they are universal things.
Starting point is 00:57:34 Yeah, I mean, that's one of the two options. The other option is that we're just not that imaginative, and so therefore we keep reinventing what we are familiar with, right? Well, but again, I think artificial life, it's one way of actually getting this, this taste that it's not us. It's probably the constraints that are there. And I want to also make the point that why in the neural networks of the brain, we see multi-layers of threshold elements.
Starting point is 00:58:07 And then when you go into it inside cells, you look at how genes interact with each other. What kind of models you make? You make models with threshold element responses and layers. and why immune system when you make a model of the network of immune cell interactions is a threshold network with sometimes multi-layers? It's a bit suspicious. Well, good. So I've heard people mention this, but you're emphasizing it more strongly.
Starting point is 00:58:36 So that's very interesting. The idea of a threshold element, you know, you have some input, but your output is not just proportional to how much input you have. It's like you get no output for a little bit of input, and then a lot of output for. for more, there's a threshold that you cross. So what is it that makes that so crucial to this kind of architecture? Why is that so good?
Starting point is 00:58:57 Why is this non-linearity so important? Well, I guess that I don't have a complete answer for that. It's something I'm thinking about. Yeah. But really a threshold element means that you can perform the simplest way of integrating signals or making a decision whether or not a majority of input signals crosses,
Starting point is 00:59:19 a given boundary. If you isolate that just from one neural receiving input from everyone else, that's not very meaningful. But if you think in the network where different parts of the network have to wait, what's the state of the system, thresholds, all elements are really, really an efficient solution. Okay. Maybe the only one. Good. Yeah. So that presumably leads us to lessons for constructing life and cognition, whether it's an AI or robots, etc. Are these, are the lessons they're clear yet, or are we still learning them? We're still learning. I mean, there's been beautiful achievement with artificial intelligence in the past, for example, in relation with language, and sometimes showing you that emergence, and I do think that is the key, that
Starting point is 01:00:12 Emergent phenomena is going to be the really relevant story here. Like many years ago, Luke Stills was working with this robots that exchange words, inventing words, and preaching agreements. And in the end, you get a situation where your robots have made a lexicon to refer to the world, which was more or less prevalent, but surprise. In order to make sense of the world, they invent a protogrammer, which is emergent. It wasn't planned. So it's kind of a, I think, an interesting insight into thinking of if we allow artificial
Starting point is 01:00:51 intelligence systems into have opportunities for, for example, having embodiment, embodiment is so important in actually reaching cognition. Maybe we will see big advances, right? But right now, the machines or the networks just leaving this, I used to say, the little kind of in a dark room with no world, right? The world doesn't exist. And they have no body, of course. Right.
Starting point is 01:01:22 Well, we have had people on the podcast talking about symbolic versus connectionist approaches to AI and the idea of the AI building a model of the world versus just trying to predict the, you know, what word comes next in the sentence. And there is this weird tension because the successful implementations have been mostly connectionist, right? You know, just huge neural networks that you feed a lot of data into and let it try to predict the next sentence. But my impression is that they don't actually have a model of the world inside. And that that seems like a kind of limitation. But I know that that's also controversial in the field. It is controversial, yes.
Starting point is 01:02:06 Yeah, I mean, they don't have a model. The thing is, as I was saying before, we cannot ignore the fact that since we do have models of the world and we can have a theory of mind, it's important to think that in the future of artificial intelligence is not going to be just what people is expecting, the real intelligence. I use this example from this movie.
Starting point is 01:02:34 I always recommend, I'm a very movie person. Robot and Frank. Okay. It's an amazing story about this person that has stats Alzheimer and the kids bring a robot. And it's clear from the beginning. The robot is not intelligent and uses natural language, which that makes a big difference
Starting point is 01:02:59 and changes because it were. But it's not intelligent, and it's a scene that I really find so fascinating, where the robot is saying, Frank, erase my memory. I don't know you don't like it, but you'd like to hear that by not a person. And that's the case, but it doesn't change the fact that, as it happens maybe with our pets, for example, where communication is actually very limited. It doesn't matter much, really, because in a way, it's kind of looking ourselves in a mirror that is evolving in time as if it was someone else. Hey, we're having weight much, what will be the implication to that? Good.
Starting point is 01:03:43 So, I guess to connect before, to what you said before about embodiment, we're going to, it won't take too long, I imagine, before we are taking the AIs that we've built. large language models and bodying them, right, and putting them in robots and giving them bodies and maybe even the scary part to me is giving them hunger and giving them desires to, you know, get resources out there in the world to persist. Is that a thing that is coming and should we be scared? I don't think we should be scared for one reason. You mentioned the right thing. We could be scary if in a way they are goals. or emotions or in a way
Starting point is 01:04:32 kinds of potential responses that have to do with something that are very human. I always find funny all this discussion about the artificial intelligence that will kill us. But my question is why? Why? Yeah. What is the motivation for that?
Starting point is 01:04:50 And for human, it is natural because as I was saying before, we can have a theory of mine, we can understand how the others think in a way, Right? We put yourself in the mind of another. And that was probably the origin of consciousness, if I can say. Selection pushes into understanding the mind of the others. You are equipped with language, the brain time machine.
Starting point is 01:05:14 For me, it's almost inevitable in the end. You understand that you are also a special individual in a world and you understand yourself. So, sorry, I'm kind of moving around. No, that's good. You should. But it's sort of charmingly optimistic. I'm probably on your side, but I guess the opposite argument would be how much risk are you willing to take? As I said, I mean, intentions is something that really has to do with a layer of commission that I think escapes completely right now for artificial intelligence. It's not there.
Starting point is 01:05:55 Right. So you don't have any kind of motivation, which is a really high-level complex category. thinking, why you should be harmful at all. Okay, so by thinking about the space of cognitions, liquid and solid brains and things like that, are we led to realize that there are possible cognitions that we haven't yet explored and can we build them either in silicon or even biologically? Well, that's something that I'm very interested for one reason. When we build this space of cognition, we have several candidates ways of drawing this.
Starting point is 01:06:36 We use cubes in evolutionary biology that are known as the morphos spaces. You try to find out axes that represent relevant properties. For example, how complex is the computational power of one of these systems, how autonomous it is. or in the vertical axis, for example, how social it is. And it's interesting that if you put there all the objects you know, which means animals from octopy to humans and ants, and also robots and AI systems, there's a big void.
Starting point is 01:07:16 There's a domain there that is empty. Why is that? It happens like in physics, that sometimes you find out there's in a space of phase page you had, this is forbidden. Is it forbidden or is something that we haven't actually observed or maybe can be engineered, right? That's all the fascinating questions that has emerged from the research. Well, we're late in the podcast, so you are allowed to speculate. What do you think?
Starting point is 01:07:46 Well, my experience from another system, which is morphogenetic systems, so we have been also, since you have a citatic biology web lab, We can actually play with some things. And one of the spaces that we created, we also had a void, a big void of morphologies that we couldn't observe. But we were able to engineer some, which means that for us, there was a path that maybe evolution was unable to follow, but we could do it.
Starting point is 01:08:17 Is it going to happen with cognitions? That's one of the most fascinating things that comes out from the research, like, you know, finding out that this is an empty place. Why is that? Right. And maybe something that we can invent or maybe it is forbidden. So maybe just for the audience a little more about the synthetic biology aspect of this. So you're going in as intelligent designers and editing the DNA and making new organisms?
Starting point is 01:08:50 Okay. Well, let's not use the term intelligent designers. Just in case. You shouldn't. Well, the thing is, synthetic biology allows us to interrogate nature in very interesting ways. I mean, it's a powerful tool for biomedical research. It's an amazing way of actually playing with living cells for us as complexity theories, actually. It's a new way of actually asking questions.
Starting point is 01:09:18 Like, for example, if I want, and this is an ongoing project, if I want to transform bacteria, modified genetically in such a way that they behave like ants. Ants can solve problems because they communicate in special ways. For example, they can find the shortest path between the nest in a given source of food. Could we trickle bacteria? And if we can, why is that? Right? Because bacteria could benefit from having extra cognition to do some things.
Starting point is 01:09:53 again, doesn't seem to be the case. It's because evolution couldn't get there or because there are tradeoffs. It's not worse to actually develop more cognition. So it's an amazing way of actually kind of creating things that solutions that nature hasn't found and see how far can you go. Yeah, good.
Starting point is 01:10:17 So plenty of work to be done. I like that. I'll give you one last thing to speculate about and so again, relating the podcast, if a colony of ants can be thought of as a collective intelligence, you know, at what point does a group of human beings become a collective intelligence? You know, is Spain conscious? Well, I think it's not. I think I will say to things. One, Edward Wilson said very well something, which is, from the point of view of a
Starting point is 01:10:52 society, we have nothing to learn from hands. Sometimes it's easy to think that it's analogies, and I would have to be very careful with that. Well, I mean, sometimes it's so disappointing to see humans behave, that it's, it's, it's, this collective intelligence, I, you think that this is not, it's not going to happen. There's clearly an amazing, an amazing thing that has to do with some collective phenomena here, which is the culture. Culture has been co-evolving with brains for humans.
Starting point is 01:11:28 And I like it when Michael Lachman, the Santa Feist, told me that we were discussing about the trade-offs that you see. It seems that the more complex is an organism in a society, the less social is the whole system, whereas society in insects is huge. The individuals turn to be more strong. stupid. But he don't look, but look at humans. I mean, if you isolate this amazing human brain from the rest is absolutely useless. It's worth nothing, right? Because isolated from culture,
Starting point is 01:12:06 from learning, from imitation, from language acquisition, what are we? Nothing is very interesting, just something to make, to make, makes you think about how culture and brains go involved. Right. So groups are not really, groups of human beings are not themselves conscious, but we do rely on their input and social interactions to make us who we are. Oh, and it's clearly being social is part of the fact that we have cooperating agents. And that was a huge part of our success as a species, despite that, as I was saying before, some days it doesn't seem to be the case.
Starting point is 01:12:45 I was going to say, I hope we can keep up that success for a little while longer. So Ricard Selday, thanks very much for being on the Mindscape podcast. Thank you very much. What if you could have even more and more and more help to pursue your goals? At LPL Financial, we offer more ways for advisors and their clients to thrive. So what if you could? Paid advertisement. Investing involves risk, including potential asset principal, LPL Financial LLC member FINRA, SIPC.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.