Theories of Everything with Curt Jaimungal - Joscha Bach Λ Ben Goertzel: Conscious Ai, LLMs, AGI

Starting point is 00:00:00 The first breakthrough to incontrovertibly human-level AGI to a superintelligence is months to years. Will that be good or bad for humanity? To me, these are less clear than what I think is the probable timeline. Josje Bak is known for his insights into consciousness and cognitive architectures, and Ben Gortzel is a seminal figure in the world of artificial general intelligence, and known for his work on OpenCog. Both are coming together here on Theories and known for his work on open cog. Both are coming together here on Theories of Everything for a theolocution.

Starting point is 00:00:34 A theolocution is an advancement of knowledge, couched both in tenderness and regard, rather than the usual tendency of debate, which is characterized by trying to be correct, even to the detriment of the other person, maybe even destructively, maybe even sardonically. We have a foray in this episode into semantics and Pierce's sign theory. This also extends into what it truly takes to build a conscious AGI. An AGI is an artificial general intelligence which mimics human-like intelligence. But then the question lingers, what about consciousness? What differentiates mere computation from awareness? Man, this was a fascinating discussion and there will definitely be a part two. Recall the system here on toe, which is if you have a question for any of the guests,

Starting point is 00:01:10 whether here or on a different podcast, you leave a comment with the word query and a colon. And this way, when I'm searching for the next part with the guest, I can just press control F and I can find it easily in the YouTube studio backend. And then I'll cite your name either aloud, verbally or in in the description to those of you who are new to this channel my name is kurt jaimungal and this is theories of everything where we explore usually physics and mathematics related theories of everything how do you reconcile quantum mechanics with general

Starting point is 00:01:38 relativity for instance that's the standard archetype of the toe but also more generally where does consciousness come in? What role does it have to play in fundamental law? Is fundamental, quote unquote, the correct philosophical framework to evaluate explanatory frameworks for the universe and ourselves? We've also spoken to Josha three times before. One solo, that episode is linked in the description. Another time with John Verveke and Josha Bach, and another time with Donald Hoffman and Josha Bach. That was a legendary episode. Also, Ben Gortzel has given a talk on this program, which was filmed at Mindfest, which was a conference about artificial intelligence

Starting point is 00:02:11 and consciousness. If you enjoy the topics of mathematics, physics, consciousness, AI, free will, and philosophy, then consider subscribing to get notified. Enjoy this episode with Josje Bak and Ben Gortzel. enjoyed discussing with ben it's uh it's always been fun and uh it's i think the first time we are on a podcast together yes wonderful so let's bring some of those off-air discussions to the forefront how did you all meet we met first at the agi conference in memphis ben had organized it and i went there because i wanted to work on ai in the traditional Minsky and sense. And I worked on a cognitive architecture. My PI didn't really like it. So I paid my own way to this conference to publish it.

Starting point is 00:03:12 And I found like-minded people there. And foremost among them was Ben. Great. What's something, Ben, that you've changed your mind about in the past six months in this field, this AGI field or AI field? And then, Josje, the question will go to you right afterward. I don't think I've changed my mind about anything major related to AGI in the last six months, but certainly seeing how well LLMs have worked over the last nine months or so has been quite interesting.

Starting point is 00:03:50 I mean, it's not that they've worked a hundred times better than I thought they would or something, but certainly just how far you can go by this sort of non-AGI system that munges together a huge amount of data from the web has been quite interesting to see. And it's revised my opinion on how much of the global economy may be converted to AI even before we get to AGI, right? So it's shifted by thinking on that a bit, but not so much on fundamentally how do you build an AGI,

Starting point is 00:04:31 because I think these systems are somewhat off to the side of that, although they may usefully serve as components of integrated AGI systems. And Joscha? Well, I have some things that changed my mind are outside of the topic of agi i thought a lot about the way in which psychology was conceptualized in greece for instance but i think that's maybe too far out here in terms of ai i looked into some kind of new learning algorithm that fascinate me and that are more brain-like and move a little bit beyond the perceptron. And I'm making slow and steady progress in this area.

Starting point is 00:05:14 It doesn't feel like there is a big singular breakthrough that dramatically changed my thinking in the last six months. But I feel that there is an area where we're beginning to understand more and more things. All right, let's get to some of the comparisons between you all, the contrasting ones. It's my understanding that, Josje, you have more of the mindset of everything is computation or all is computation, and Ben, you believe there to be other categories. I believe you refer to them as archetypal categories, or I may have done that. And I'm unsure if this is a fair assessment, but please elucidate me. I think that everything that we think happens in some kind of language. And perception also happens in some kind of language. And a language cannot refer to anything outside of itself. And in order to be semantically

Starting point is 00:06:03 meaningful, a language cannot have contradictions. It is possible to use a language where you have figured out how to resolve all the contradictions, as long as you have some hope that there is a way to do it. But if a language is self-contradictory, its terms don't mean anything. And the languages that work, that we can use to describe anything, any kind of reality and so on, turn out to be representations that we can describe via state transitions. And a number of ways in which we can conceptualize systems that are doing state transitions, for instance, we can think about whether they're deterministic or indeterministic, whether they're linear or branching. whether they are linear or branching. And this allows us to think of these representational languages as a taxonomy,

Starting point is 00:06:53 but they all turn out to be constructive. That means modern Palance computational. There was a branch of the mainstream of mathematics was not constructive before Gödel. That means language of mathematics allowed to specify things that cannot be implemented. And computation is the part that can be implemented. I think for something to be existent, it needs to be implemented in some form. And that means we can describe it

Starting point is 00:07:17 in some kind of constructive language. That's basically the sword Rhinos. It has to do with epistemology. And the epistemology determines the metaphysics that I can have. Because when I think about what reality is about, I need to do this in a language in which my words mean things. Otherwise, what am I talking about? What am I pointing at? When I'm pointing at, I'm pointing at the representation that is basically a mental state that my own mind represents and projects into

Starting point is 00:07:46 some kind of conceptual space or some kind of perceptual space that we might share with others. And in all these cases, we have to think about representations. And then I can ask myself, how is this representation implemented in whatever substrate it is? And what does this signify about reality? And what is reality signify about reality? And what is reality and what is significance? And all these terms turn out to be terms that, again, I need to describe in a language that is constructive, that is computational. And in this sense, I am a strong computationalist, because I believe that if we try to use non-computational terms to describe reality, and it's not just because we haven't gotten around to formalizing them yet,

Starting point is 00:08:27 but because we believe that we found something that is more than this, we are fundamentally confused and our words don't mean things. And Ben? I think that, yeah, I tend to start from a different perspective on all this philosophically. I mean, I think there's one minor technical point I feel a need to quibble with than what Joshua said, and then I'll try to outline my point of view from a more fundamental perspective. I mean, the point I want to quibble with is, it was stated that if a logic or language contains contradictions, it's meaningless. I mean, of course, that's not true. There's a whole discipline of paraconsistent logics,

Starting point is 00:09:19 which have contradictions in them and yet are not meaningless. And there are constructive paraconsistent logics. And you can actually use, you know, Curry-Howard transformations or operational semantics transformations to map paraconsistent logical formalisms into gradually typed programming languages and so forth. So, I mean, contradictions are not necessarily fatal to having meaningful semantics to a logical or computational framework, and this is something that's actually meaningful in my approach to AGI on the technical level, which we may get into later. But I want to shift back to the foundation of life, the universe, and everything here. So I mean, I tend to be phenomenological in my approach, more so than starting from a model of reality. And

Starting point is 00:10:21 these sorts of things become hard to put into words and language, because once you project them into words and language, then yeah, you have a language, because you're talking in language, right? But talking isn't all there is to life. It isn't all there is to experience. And I think the philosopher Charles Peirce gave one fairly clear articulation of some of the points I want to make. You could just as well look at Lao Tzu, or you could look at the Vedas or the book Buddhist Logic by Stravatsky, which gives similar perspectives from a different cultural background. So, if you take Charles Peirce's point of view, which at least is concise, he distinguishes a number of metaphysical categories. I don't follow him exactly, but let me start with him. So he starts with

Starting point is 00:11:17 first, by which he means qualia, like raw, unanalysable,, just it's there, right? And then he conceives second, by which he means reaction, like billiard ball bounces off each other. It's just one thing is reacting to something else, right? And this is how he's looking at the crux of classical physics, let's say. Then by what Peirce calls third, he means relationships. So, one thing is relating to other things. And one of the insights that Charles Peirce had writing in the late 1800s was that once you can relate three things, you can relate four, five, six, ten, any large finite number of things, which was just a version of what's very standard now, of reducing a large number of logical relations to sort of triples or something, right? So, Peirce looked

Starting point is 00:12:14 at first, second, and third as fundamental metaphysical categories, and he invented quantifier logic as well with a for all and there exists in quantifier binding so he as purse would look at it computation and logic are in the realm of third and if you're looking in that metaphysical category of third then you say well everything's a relationship on the other hand if you're looking for within from within the metaphysical category of second you're looking at it like well everything's just reactions if you're looking at it from within the metaphysical category of second, you're looking at it like, well, everything's just reactions. If you're looking at it from within the metaphysical category of first, then it's like, whoa, it's all just there, right? And you could take any of those points of view and it's valid in itself. Now, you could extend beyond Persis categories. You could say, well, I'm going to be

Starting point is 00:13:01 a Zen Buddhist and have a category of zero, like the unanalyzable pearly void, right? Or you could say, well, I'm going to be a Zen Buddhist and have a category of zero, like the unanalysable pearly void, right? Or you could go Jungian and say, okay, these are numerical archetypes, one, two, three, but then we have the archetype of four, which is sort of synergy and emergence. It's sort of mandalic. Yeah, so what I was saying is Peirce had these three metaphysical categories, which he viewed as just ontologically, metaphysically distinct from each other. So what Chalmers would call the hard problem of consciousness in Persian language is like, how do you collapse third to first? And Peirce would be just like, well, you don't. They're different categories. You're an idiot to think that you

Starting point is 00:13:43 can somehow collapse one to the other. So in that sense, he was a dualist, although more than a dualist, because he had first, second, and third. Now, I think you could go beyond that if you want. You could go Zen Buddhist and say, well, we have a zero category of the original, ineffable, self-contradictory, pearly void. And then you have the question of, is zero really the same as one, which is like the Zen Buddhist paradox of non-dualism and so forth in a certain form.

Starting point is 00:14:11 You can also go above Persis' three metaphysical categories, and you can say, okay, well, why not four, fourth? Well, to Carl Jung, four was the archetype of synergy, and many mandalas were based on this fourfold synergy. Why not five? Well, five, you have the archetype of synergy, and many mandalas were based on this fourfold synergy. Why not five? Well, five, you have the fourfold synergy, and then the birth of something new out of it, right? So, I can see that the perspective of third, the perspective of computation, is substantially where you want to focus if you're engineering an AGI system, right? Because you're writing a program, and the program is a set of logical relationships. The program is written in a language. So I don't have any disagreement that this is the focal point when you're

Starting point is 00:14:59 engineering an AGI system. But if I want to intuitively conceptualize the AGI's experience, I don't feel a need to try to reduce the whole metaphysical hierarchy into third, just because the program code lives there. I mean, this is sort of a... It's not so much about AI or mathematical or computational formalism i mean these are just different philosophical perspectives which it becomes arduous to talk about because natural language terms are are imprecise and ambiguous and slippery. And you could end up spending a career

Starting point is 00:15:48 trying to articulate what is really meant by relationship or something. All right, Joscha. I think it comes down to the way in which our thinking works and what we think thinking is. You could have one approach that is radically trying to build things from first principles and when we learn how to write computer programs this is what we might be doing when i

Starting point is 00:16:15 started programming i had a commodore 64 i was a kid i didn't know how to draw a line commodore 64 is basic doesn't have a command to draw a line what you need to draw a line, Commodore 64 is basic, doesn't have a command to draw a line. What you need to draw a line in a Commodore 64 is you need to learn a particular language. And this language, in this case, is basic. You can also learn Assembler directly. But it's not hard to see how Assembler maps to the machine code of the computer. And the machine code works in such a way that you have a short sequence of bits organized into groups of eight bytes. And these bytes are interpreted as commands by the computer.

Starting point is 00:16:52 They're basically like switches or train tracks. You could imagine every bit determines whether a train track goes to the left or to the right. And after you go through eight switches, you have 256 terminals where you can end, right? So if you have two options to switch left or right, and each of these terminals, you have a circuit, some kind of mechanism that performs a small change in the computer. And these changes are chosen in such a way that you can build arbitrary programs from them. And when you want to make a line, you need to learn a few of these constructs that you use to manipulate the computer. And first of all, in the Commodore 64, you need to write a

Starting point is 00:17:31 value on a certain address that corresponds to a function on the video chip of the computer. And this makes the video chip forget how to draw characters on screen and instead interpret a part of the memory of the computer as pixels that are to be displayed on the screen. And then you need to tell it which address in working memory you want to start by writing two values into the graphic chip, which are encode for a 16-bit address in the computer. And then you can find the bits in your working memory that correspond to pixels on the screen.

Starting point is 00:18:06 And then you need to make a loop that addresses them all in order, and then you can draw a line. And once I understood this, I basically had a mapping from an algebraic equation into automata. That is what the computer is doing. It's an automaton at the lowest level that is performing geometry. And once you can draw lines, you figure out also how to draw curved shapes, and then you can draw 3D shapes, and you can easily derive how to make that.

Starting point is 00:18:34 And I did these things as a kid, and then I thought the mathematicians have some kind of advanced way, some kind of way in which I deeply understand what geometry is in ways that goes far beyond what I am doing. And mathematics teachers had the same belief. They basically were gesturing at some kind of mythological mountain of mathematics where there was some deep and scrutable knowledge on how to do continuous geometry, for instance. And it was much, much later that I started to look at this mountain and realized that it was doing the same thing that I did on my Tacoma Dora 64, just with Greek notation. And

Starting point is 00:19:11 there's a different tradition behind it, but it was basically the same code that I've been using. And when I was confronted with notions of space and continuous space and many other things, I was confronted with a conundrum. I thought I can do this in my computer and it looks like it, but there can be no actual space because I don't know how to construct it. I cannot make something that is truly continuous. And I also don't observe anything in reality around me that is fundamentally different from what I can observe in my computer to the degree that I can understand and implement

Starting point is 00:19:43 it. So how does this other stuff work? And so imagine somebody has an idea of how to do something in a way that is fundamentally different from what could be in principle done in computers. And I ask them how this is working. It goes into hand waving. And then you point at some proofs that have been made that show that the particular hand waving that they hope to get to work does not

Starting point is 00:20:05 pan out and then i hope there is some other solution to make that happen because they have the strong intuition and i asked where does this intuition come from how did it actually get into your brain and then you look at how does the brain work there is firing between neurons there is interaction with sensory patterns on the systemic interface to the universe, how were they able to make inferences that go beyond the inferences that I can make? This is one way of looking at it. And then on the other end of the spectrum, and this one is more or less in the middle, there is degraded form of epistemology,

Starting point is 00:20:38 which is you just make noises, and if other people let you get away with it, you're fine. So you just make sort of grunts and hand-wavy movements and you try to point at things and you don't care about anything if it works and if a large enough group of high-status people is nodding, you're good. And this epistemology of what you can get away with doesn't look very appealing to me because people are very good at being wrong in groups.

Starting point is 00:21:02 doesn't look very appealing to me because people are very good at being wrong in groups. Yeah, I mean, saying that the only thing there is is language because the only thing we can talk about in language is language. I mean, this is sort of tautologous in a way, right? No, no, that's not quite what I'm saying. I'm not saying the only thing there is language, of course. Language is just a representation.

Starting point is 00:21:27 It's a way to talk about things and to think about things and to model things. And obviously not everything is a model, just everything that they can refer to as a model. And so there is... That, that, that, that... I mean.

Starting point is 00:21:47 Razor blades are like diving boards. The longer the board, the more the wobble, the more the wobble, the more nicks, cuts, scrapes. A bad shave isn't a blade problem. It's an extension problem. Henson is a family owned aerospace parts manufacturer that's made parts for the International Space Station and the Mars rover. Now they're bringing that precision engineering to your shaving experience. By using aerospace-grade CNC machines, Henson makes razors that extend less than the thickness of a human hair. The razor also has built-in channels that evacuates hair and cream,

Starting point is 00:22:20 which make clogging virtually impossible. Henson Shaving wants to produce the best razors, not the best razor business. So that means no plastics, no subscriptions, no proprietary blades, and no planned obsolescence. It's also extremely affordable. The Henson razor works with the standard dual edge blades that give you that old school shave with the benefits of this new school tech. It's time to say no to subscriptions and yes to a razor that'll last you a lifetime. Visit hensonshaving.com slash everything. If you use that code, you'll get two years worth of blades for free. Just make sure to add them to the cart. Plus 100 free blades when you head to h-e-n-s-o-n-s-h-a-v-i-n-g.com slash everything and use the code everything.

Starting point is 00:23:02 v i n g dot com slash everything and use the code everything you can't know that right you can hypothesize that but you can't know that and this gets into I guess it depends what you mean by that

Starting point is 00:23:18 I cannot know anything that I cannot express I can know many things I can't express in language but I mean that's just I can't express in language. But I mean, that you're staring into someone's eyes and you have a deep experience that you're seeing that person and you're just sharing a shared space of experience and being, I mean, in that moment, that is something you both know you're not going to be able to communicate fully in language and it's experientially there. Now, Buber wrote a bunch of words about it, right? And those words communicate something special to me

Starting point is 00:24:13 and to some other people. But of course, someone else reads the words that he wrote and says, well, you are merely summarizing some collection of firings of neurons in your brain in some strange way, deluding yourself that it's something else, right? So, I mean, I think from within the domain of computation and science, the domain of computation and science, you can neither prove nor disprove that there exists something beyond the range of computation and science. And if you look at scientific data, I mean, the whole compendium of scientific data ever gathered by the whole human race is one large finite bit set, basically. I mean, it's a large set of data points with finite precision to each piece of data.

Starting point is 00:25:10 So, I mean, it might not even be that huge of a computer file if you try to assemble it all, like all the scientific experiments ever done and agreed by some community of scientists. So you've got this big finite bit set, right? And then science, in a way way is trying to come up with you know concise reasonable looking culturally acceptable explanations for this huge finite bit set that can be used to predict outcomes of other experiments and which finite collection of bits will will emerge from those other experiments in a way that's accepted by a certain community.

Starting point is 00:25:49 Now, that's a certain process. It's a thing to do. It has to do with finite bit sets and computational models for producing finite bits, right, and the finite sets of bits. And that's great. Nothing within that process is going to tell you that that's all there is to the universe or that that isn't all there is to the universe. I mean, it's a valuable, important thing. Now, to me, as an experiencing mind,

Starting point is 00:26:22 I feel like there's a lot of steps I have to get to the point where I even know what a finite bit set is, or where I even know what a community of people validating that finite bit set is really, or what a programming language is. So I mean, I keep coming back to my phenomenal Hodgkin experience. First, there's this field of nothingness or contradictory nothingness that's just floating there. Then some indescribable forms flicker and emerge out of this void. And then you have some complex pattern of forms there, which constitutes a notion of a bit set or an experiment or a computation. And from this

Starting point is 00:27:06 phenomenological view, by the time you get to this business of computing and languages, you're already dealing with a fairly complex body of self-organizing forms and distinctions that popped out of the void. And then this conglomeration of forms that in some enough of a way has emerged out of the void is selling no i am i am everything the only thing that exists in a fundamental sense is what is inside me and i mean you can't if you're inside that thing you can't you can't refute or really demonstrate that but again from from anGI view, it's all fine, because when we talk about building an AGI, what we're talking about is precisely

Starting point is 00:27:51 engineering a set of computational processes. Like, I don't think you need to do, like, you don't need some special first tronium to drop into your computer to give the AGI the fundamental qualia of experience or something. There are two points now. Let me briefly interject

Starting point is 00:28:14 so we don't forget. Let's just allow Josje to speak because there are quite a few threads and some may be dropped. Also, it appears as if you're using different definitions of knowledge. If we use this traditional philosophical notion of justified true belief, it means that I have to use knowledge in a context where I can hope to have a notion of what's true. So, for instance, when I look at your face and experience a deep connection with you, and I report I know we have

Starting point is 00:28:40 this deep connection, I'm not using the word know in the same sense. What I am describing is an observation. I'm observing that I seem to be looking at a face and I'm observing that I have the experience of having a deep connection. And I think I can hope to report on this truthfully. But I don't know whether it's true that we have that deep connection. this truthfully. But I don't know whether it's true that we have that deep connection. I cannot actually know this. I can make some experiments to show how aligned we are and how connected we are and so on to say this perception or this imagination has some veracity. But here I'm referring to a set of patterns. There are dynamic patterns that i perceive and then there is stuff that i can reflect on and disassemble and talk about and convey and model and this is a distinct category in the

Starting point is 00:29:33 sense it's not in contradiction necessarily what you're saying it's just using the word knowing in different ways is implied here because i can relate the pattern to you that I'm observing or that I think I'm observing. But this is a statement about my mental state. It's not a statement about something in reality about the world. And to make statements about the world, I probably need to go beyond perception. The second aspect that we are now getting to is when you say that reality and minds might have properties that are not computational,

Starting point is 00:30:06 yet your AGI is entirely computational and doesn't need any kind of first principles wonder machine built into it that goes beyond what we can construct from automata. Are you establishing that AGIs, artificial general intelligence with potentially superhuman capabilities, are going to still lagging behind what your mind is capable of? No, not at all. I just think the other aspects are there anyway and you don't need to build them. So you're going to make the non-computational parts of reality using computation? So you're going to make the non-computational parts of reality using computation?

Starting point is 00:30:47 No, you don't have to make them. They're already there. I mean, if you take just a more simple point of view where you're thinking about first and third, and Peirce was basically a panpsychist, right? So he believed that matter is mind, bound with habit as he said he he believed that every little particle had its own spark or element of of consciousness and and awareness in it so i mean from that standpoint i mean this this kind of bubbly water that i'm holding up has its own variety of conscious awareness to it, which has different properties than the conscious awareness in my brain or yours.

Starting point is 00:31:29 So from that standpoint, if I build an AGI program that has something around the same patterns and structures and dynamics as a human brain and as the sort of computational aspect of the human mind, from that standpoint then most likely the same sort of firstness, the same species of subjective awareness will be associated with that AGI machine that you've built. But it's not that you needed to construct it. I mean, any more than you need to explicitly construct the positioning in time of your computer or something. You build something, it's already there in time. You don't have to build time.

Starting point is 00:32:17 I mean, you just build it and it's there in time. You didn't need a theory of time, and you didn't need to screw together moment t to moment t plus one either. The perspective is more that awareness is ambient and it's there, you don't need to build it. Of course there's subtlety that different sorts of constructions, may have different sorts of awareness associated with them, and there's philosophical subtlety in how you treat different kinds of firsts when you're operating at a level where relationship doesn't exist yet, right? In what sense is the experience of red different from the experience of blue, even though articulating that difference already brings you into the realm of third, right?

Starting point is 00:33:11 And this gets back to, it gets back to non-duality and a bunch of stuff that Peirce wrote hundreds of pages about, right? I haven't read these pages, so I don't really understand them. I think it's conce that uh particles are conscious or intelligent uh but this would require that they have or imply they have more complicated causal structure than the computer that i'm currently using to communicate with you and by that's possible i seem it seems to me that it's uh that there are simpler ways in which particles could be constructed to do the things that they are doing it seems to me me that there are simpler ways in which particles could be constructed

Starting point is 00:33:45 to do the things that they are doing. It seems to me sufficient that there are basically emergent error-correcting codes on the quantum substrate and would just emerge over the stuff that remains statistically predictable in a branching multiverse. I don't need to be conscious to do anything like that. Maybe

Starting point is 00:34:05 if we do more advanced physics, we figure out, oh no, this error correcting code that just emerges similar to a vortex emerges in the bathtub when you move your hand and only the vortex remains and everything else dissipates in the wave background that you are producing in the chaos

Starting point is 00:34:21 and turbulences. It could be, to me, possible that particles are like this. They're little stable twirls, vortices, that self-stabilize after the non-stable stuff is dissipating. And to achieve this, I don't think that I need to posit that they are conscious. It could be that I figure out, oh, no, this is not sufficient.

Starting point is 00:34:43 We need way more complicated mass and structure to make this happen. So they need some kind of coherence improving operator that is self-reflexive and eventually leads to the structure. Then I would say, yeah, maybe this is a theory that we should seriously entertain. Until then, I'm undecided and Occam's razor says I can construct what I observe at this level of elementary particles atoms and so on by assuming that they don't have any of the conscious functionality that exists in my own mind and the other way would be you can redefine the notion of consciousness into some principle of self-organization it is super basic but this would redefine consciousness into something else because there is a lot of self-organizing stuff that does not fall into the same category that an anesthesiologist makes go

Starting point is 00:35:29 away when he gives you an anesthetic and to me consciousness is that thing which seems to be suspended when you get an anesthetic and that stops you from learning and currently interacting with the world i mean that at least gives me a chance to repeat once again my favorite quote from Bill Clinton, former U.S. president, which is, that all depends what the meaning of is is. That's interesting, Ben, because the first time, Yoshua, I don't know if you remember, I brought up that quote. I don't remember the context, but I said, yeah, that also depends on what is, is. The question is, what do you mean by is, right? So this is the story. It sounds like Bill Clinton.

Starting point is 00:36:13 It depends upon what the meaning of the word is. The previous podcast with Josje Bak as a solo episode is linked in the description, as well as the previous podcast with Ben solo as well as Yoshibach with Donald Hoffman and Yoshibach with John Verveke. Every link as usual to every source mentioned in this podcast on every single Toe podcast is in the description.

Starting point is 00:36:36 Yeah, I mean a couple reactions there and I feel I may have lost something in the buffering process there but I feel I may have lost something in the buffering process there. But I think that, let me see, so that, First of all, about causality and firstness or raw experience, I mean, almost by definition of how Peirce sets up his metaphysical categories, I mean, a firstness doesn't cause anything.

Starting point is 00:37:21 So you're not going to come up with a case where, cause anything. So you're not going to come up with a case where I need to assume that this particle has experience or else I can't explain why this experiment came out this way. I mean, that would be a sort of category error in Peirce's perspective. So, if the only sort of thing you're willing to attribute existence to is something which has a demonstrable causal impact on some experiment, then by that assumption, I mean, that's essentially equivalent to the perspective you're putting forth, that everything is computation. And Peirce didn't think other categories besides third were of that nature. Now, there's also a just shallow semantic matter tied into this, which is the word consciousness is just highly ambiguous. So, I mean, Yosha, you seem to just assume that human-like consciousness is

Starting point is 00:38:34 consciousness, and I don't really care if people want to reserve the word consciousness for that. Then we just need some other word for the sort of ambient awareness and everything in the universe right so that there's lengthy debates among academics i'm like okay do we say a particle is conscious or do we say it's proto-conscious right so then you can say okay we have proto-consciousness versus consciousness or we have have raw consciousness versus reflexive consciousness, or human-like consciousness. I spent a while reading all this stuff. I wrote some things about it. In the end, I'm just like, this is a game that overly intellectual people are playing to entertain themselves. It doesn't really matter. I've got my experience of the universe.

Starting point is 00:39:25 I know what I need to do to build AGI systems. And arguing about which words to associate with different flavors and levels of experience is you're just kind of running around in circles. Conceptually, it's an important question. Is this camera that is currently filming my face and representing it and then relaying it to you aware of what it's doing? And is this just a matter of degree with respect to my consciousness? Is this representing some kind of ambient awareness

Starting point is 00:39:54 of the universe? And are particles doing the same thing? And so on. These are questions that I think I can answer. And if I don't answer them, my thinking will become so mushy that my thoughts are meaningless, and I will not be able to construct anything. I mean, if the only kind of answer that you're interested in are rigorous scientific answers, then you have your answer by assumption, right? I mean, answering questions by assumption is fine, It's practical. It saves our time. I think that's what you're doing. I don't see how you're not just trying to answer by assumption.

Starting point is 00:40:36 You posit that elementary particles are conscious. Then I point out that we normally reserve the word consciousness for something that is really interesting and fascinating and shocking to us, and that it would be more shocking if it projected into the elementary particles. And then you say, okay, but I just mean ambient awareness. Now we have to disassemble what ambient awareness actually means. What is aware here? What does this awareness come down to? aware here? What does this awareness come down to? And I think that you're pointing at something that I don't want to dismiss. I want to take you seriously here. So maybe there is something to what you are saying, but you're

Starting point is 00:41:12 not getting away with simply waving at it and saying that this is sufficient to explain my experience and I'm no longer interested to make my words mean things. Because they cannot communicate otherwise. We will not be able to see what idea you actually have,

Starting point is 00:41:30 what idea you're trying to convey, and how it relates to ideas that other people might have. And I'm not pointing at the institutions of science here which don't agree on what consciousness is and for the most part don't care. This is more a philosophical question here. And it's also one that is an existential question that we have to negotiate among the two or the three of us.

Starting point is 00:41:50 Yeah, let's just let Ben respond. I mean, I guess rightly or wrongly, as a human being, I've gotten bored with that question in the same way that, like, I couldn't say it's worthless at some point maybe you could convince someone like i i know people who were convinced by materials they read on the

Starting point is 00:42:11 internet to give up on on mormonism or scientology right so i can't say it's worthless to debate these points with people who are are heavily attached to an ideology I think is silly. On the other hand, I personally just tend to get bored with repeated debates that go over the same points over and over again. If I had an infinite number of clones, then I wouldn't. I guess one of the things that I get worn out with is people claiming my definition of this English word is the right one, and your definition is the wrong one.

Starting point is 00:42:58 And I guess you weren't really doing that, Joshua, but it just gave me a traumatic memory of too many things. I'm sorry for triggering you here. I'm not fighting about words. I don't care which words you're using. So when I think about an experience of what it's like and associate that with consciousness or a system that is able to create a now,

Starting point is 00:43:22 a perception of a now, then I'm talking about a particular phenomenon that I have in mind, and I would like to recreate if I can, and I want to understand how it works. And so for me, the question of whether I project this property into arbitrary parts of what I consider to be reality is important.

Starting point is 00:43:47 I understand if it's not interesting to you, and I won't force you into any discussion that would make you drop out of your particular Mormonism. I'm happy with you being a Mormon. Let me tell you how I'm looking at anesthesia, which is a concrete, specific example that's not that trivial right because that that i've only been under anesthesia once which i have wisdom teeth removed so what was it wasn't that bad but other people have had far more traumatic things done when they're under anesthesia and there there is there's always the nagging, like, since we don't really know how anesthesia works in any fundamental depth and also don't really know how the brain generates our usual everyday states of consciousness in enough depth, it's always possible that while you're under anesthesia, you're actually, in some sense, some variant of you is feeling that knife slicing through you and maybe just the memory is being cut off right and then then once you once you come back you don't

Starting point is 00:44:53 you don't you don't you don't remember it but then that might not be true but then you have to ask well okay say then you know while while my jaw is being cut open by that knife, does the jaw feel it, right? Like, does the jaw hurt while it's being sliced up by the knife? Like, is the jaw going, ah! Well, on the other hand, you know, the global workspace in your brain, like the reflective theater of human-like consciousness in your brain, may well be disabled by the anesthetic. So the way I personally look at that is, I suspect under anesthesia, your sort of reflective theater of consciousness is probably disabled by that anesthetic. I'm not 100% sure, but I think it's probably disabled, which means there's probably not like a version of Ben going,

Starting point is 00:45:51 ah, wow, this really hurts, this really hurts, and then forgetting it afterwards. So, I mean, maybe you could do that, just like disable memory recording, but I don't think that's what's happening. On the other hand, I think the jaw is having its own experience of being sawed open while you're getting that wisdom tooth removed under general anesthesia. Now, I think it's not the same sort of experience exactly as the reflective theater of consciousness that knows itself as Ben Go as Ben Goertzel is having, like the Ben Goertzel can conceptualize that it's, that it's experiencing pain. It can go like,

Starting point is 00:46:31 ow, that really hurts. And then the thinking that saying that really hurts is different than the, that which really hurts, right? There, there's many levels there, but I do think there's some sort of raw feeling that the jaw itself is,

Starting point is 00:46:44 is, is having, like even if it's not connected to that reflective theater of awareness in the brain. Now, the jaw is biological cells, so some people would agree that those biological cells have experience, biological cells have experience, but they would think like, you know, a brick when you smash it with an axe doesn't. But I suspect the brick also has that some elementary feeling. So, I mean, I think it is like something to be a brick that's smashed in half by an axe. On the other hand, it's not like something that can reflect on what it is to be a brick smashed in half by an axe.

Starting point is 00:47:33 So, I mean, that is how I think about it. But again, I don't know how to make that science because I can't ask my jaw what it feels like because my jaw doesn't doesn't speak language and even if i was able to like wire my brain into the jaw of someone else who's going through wisdom tooth removal under anesthesia like i might say like through that wire i can feel by an eye thou experience like i can feel the pain of that jaw being sliced open. But, I mean, you can tell me I'm just hallucinating that, and my own brain is improvising that based on the signals that I'm getting.

Starting point is 00:48:15 I'm not sure how you really pin that down in an experiment. Let me try. So there have been experiments about anesthesia. And I'm not an expert on anesthesiology. So I asked everybody for forgiveness if I get things wrong. But there have been different anesthetics. And some of them work in very different ways. And there is indeed a technique that basically works by giving people a muscle relaxant so they cannot move and giving

Starting point is 00:48:47 them something that inhibits deformation of long-term memory so they cannot remember what happened to them in that state. And there have been experiments that surgeons did where they were applying a tourniquet to an arm of the patient so the muscle relaxant didn't get into the arm and they could still use the arm. And then in the middle of the surgery, they asked the person that was there, lying fully relaxed and incommunicado, to raise their arm, raise their hand,

Starting point is 00:49:16 if they were conscious and aware of what was happening to them. And they did. And when they were asked if they had unbearable pain, they also raised their hand. And after the surgery, they had forgotten about it. I also noticed the same thing on surgery. I had a number of big surgeries in my life and there is a difference between different types of surgery.

Starting point is 00:49:38 There is one type of surgery where I wake up and feel much more terrified and violated than I do before the surgery. And I don't know why, because I have no memory of what happened. Also, my memory formation is impaired. So when I am in the ER and ask people how it went, I might have that same conversation multiple times, word for word, because I don't remember what they said or that I asked them.

Starting point is 00:49:59 There is another type of anesthesia, and I observed this, for instance, in one of my children and where the child wakes up and says oh the anesthesia didn't work and it was an anesthesia with gas so the child choked on the gas and you see your child lying there completely relaxed and sleeping and then waking up and starting to choke and then telling you the anesthesia didn't work. There is a complete gap of eight hours in the memory of that child in which the mental state was somehow preserved. Subjectively, the child felt a complete continuation and then was looking around and reasoning that the room was completely different,

Starting point is 00:50:37 time was very different, led to confusion and reorientation. But so I would suspect that in the first case, it is reasonable to assume or to hypothesize at least that consciousness was present, but we don't recall what happened in this conscious state. Whereas in the second one, there was a complete gap in the conscious experience and consciousness resumed after that gap. And we can test this, right? There are ways, regardless of whether we agree with this particular thing or whether we think anesthesia is important, in principle, we can perform such experiments and ask such questions. And then on another level, when we talk about our own consciousness,

Starting point is 00:51:20 there's certain behavior that is associated with consciousness that makes it interesting. Everything, I guess, only becomes interesting due to some behavior, even if the behavior is entirely internal. If you are just introspectively conscious, it still matters if I care about you. And so this is a certain type of behavior that we still care about. For instance, if I ask myself, is my iPhone conscious? The question is, what kind of behavior of the iPhone corresponds to that? And I suspect if I turn off my iPhone or smash it, it does not mean anything to the iPhone.

Starting point is 00:51:54 There is no what-its-likeness of being smashed for the iPhone. There could be a different layer where this is happening, but it's not the layer of the iPhone. Now, let's get to a slightly different point. This question of whether your jaw knows anything about being hurt. So imagine that there is surgery on your jaw, like with your wisdom teeth. Is there something going on that is outside of your brain that is processing information in such a way that your jaw could become sentient in the sense that it knows what it is and how it relates to reality,

Starting point is 00:52:25 at least to some degree and level. And I cannot rule this out. But the cells, these cells can process information. They can send messages to their neighbors and the patterns of their activation, who knows what kind of programs they can compute. But here we have a means and a motive.

Starting point is 00:52:40 The means and motive here are it would be possible for the cells to exchange conditional matrices to perform arbitrary computations and build representations about what's going on. And the motive would be that it's conceivable that this is a very useful thing for biological tissues to have in general. And so if they evolve for long enough, and it is in the realm of evolvability that they perform interactions with each other that lead to representations of who they are and what they're doing, even though they're much slower than what's

Starting point is 00:53:09 happening in our brain and decoupled from our brain in such a way that we cannot talk to our jaw. It's still conceivable, right? I wouldn't rule this out. It's much harder for me to assume the same thing for elementary particles because I don't see them having this functionality that cells have. Cells are so much more complicated that it just fits in that they would be able to do this. And so I would make a distinction. I would not

Starting point is 00:53:33 rule out that multicellular organisms without brains could be conscious but at different timescales than us requiring very different measuring mechanisms because their signal processing is probably much slower and it takes longer for them to become coherent at scale because it takes so long for signals to go back and forth if you don't have nerves. But I don't see the same thing happening for elementary particles.

Starting point is 00:53:55 I don't rule it out again, but you would have to show me some kind of mechanism. I mean, if you're going to look at it that way, which isn't the only way that I would look at it, but if you're going to look at it that way, which isn't the only way that I would look at it, but if you're going to look at it that way, I don't see why you wouldn't say the various elementary particles, which are really distributed like amplitude distributions, right? I don't know why you wouldn't say these various interacting amplitude distributions are exchanging quantum information with a with a motivation to to achieve stationary action given given their their context right i mean i mean you could tell that you can tell that story that's sort of the story that that that physics tells you they're swapping

Starting point is 00:54:39 information back and forth trying trying to make the action stationary. Yes, but for the most part, they don't form brains. They also do form brains. So elementary particles can become conscious in the sense that they can form brains, nervous system, maybe equivalent information processing architectures in nature. I just feel like you're privileging a certain level and complexity of organization because it happens to be ours. And I mean,

Starting point is 00:55:07 we have a certain level and complexity of organization and, and of consciousness. And, and I mean, a cell in my jaw has a lower one, a brick has a lower one, element protocol has a lower one. The future AGI may have a much higher one from whose

Starting point is 00:55:25 perspective our consciousness appears more analogous to a brick than to itself. I wouldn't say lower or higher. I would say that if my jaw is conscious, there are far less cells involved in my brain and the interaction between them is slower. between them is slower. So if it's conscious, it's probably more at the level of, say, a fly than a level of a brain.

Starting point is 00:55:52 And it's probably going to be as fast as a tree and in the way in which it computes rather than as fast as your brain. And I don't think that's something that is assigning some undue privilege to it. I'm just observing a certain kind of behavior. and then I look for the

Starting point is 00:56:06 means and motive behind that behavior. And then I try to construct causal structure, and I might get it wrong, there might be missing, but it's certainly not because I have some kind of speciesism that assigns higher consciousness to myself because it's me. Alright, yeah, I mean, I don't know what your

Starting point is 00:56:22 motivations are. Kurt, I have a higher level comment, which is, we're like an hour through this conversation, probably halfway through. I feel like the philosophy, the hard problem of consciousness, the hard problem of consciousness is an endless rabbit hole. It's not an uninteresting one. is an endless rabbit hole. It's not an uninteresting one. I think it's also not the topic on which Joshua and I have the most original things to say. I think each of our perspectives here are held by many other people.

Starting point is 00:56:55 I might interject a little bit. One of our most interesting disagreements is in Ben being a panpsychist and me not knowing how to formalize panpsychism in a way that makes it different from box standard functionalism. And so I do value this discussion and don't think it's useless.

Starting point is 00:57:12 But I basically feel that on almost everything else, we mostly agree except for crypto. Okay. Yeah, to me, that's almost a Zen thing. It's like, I don't know how to formalize the notion that there are things beyond all formalization. It's so fascinating to look at your frozen interlocutor. I don't know if he can see. Is he still conscious?

Starting point is 00:57:32 You can rest in that forever. Yeah, all right. Again, comparing your views, it seems like, Yosha, you're more of the mind that LLMs or deep neural nets are on or a significant step toward AGI, maybe even sufficient with enough complexity. And Ben, I think that you disagree. Yeah, I think most issues in terms of the relation between LLMs and AGI, we actually probably agree on quite well.

Starting point is 00:58:11 But I mean, obviously, large language models are an amazing technology. Like from an AI application point of view, they can do all sorts of fantastic and tremendous things. I mean, it sort of blew my mind how smart GPT-4 is. It's not the first time my mind has been blown by an AI technology. I mean, my mind was blown by computer algebra systems when they first came out. And you could do integral calculus with arbitrary complexity.

Starting point is 00:58:43 And when deep blue beat chess with just game trees i'm like whoa so i mean i don't i don't think it's the only amazing thing to happen in the history of ai but it's a it's an amazing thing like it's a big breakthrough and it's it's super cool i think that if deployed, this sort of technology could do a significant majority of jobs that humans are now doing on the planet, which has big economic and social implications. these algorithms are representing knowledge internally is not what you really need to make a full-on human level AGI system. So, I mean, when you look at what's going on inside the transformer neural network, I mean, it's not quite just a big weighted hash table of particulars, but to me it does not represent abstractions in a sufficiently flexibly manipulable way to do the most interesting things that the human mind does. And this is a subtle thing to pinpoint in that, say,

Starting point is 01:00:08 something like Othello GPT does represent abstractions. It's learning an emergent representation of where the board is, but it's learning an emergent representation of features like a black square is on this particular board position or a white square is on this particular board position. So, examples like that show that LLMs can, in fact, learn abstract representations and can manipulate them in some way, but it's very limited in that regard. I mean, in that case, it's seen a shitload of Othello games, and that's a quite simple thing to represent. So I think when you look at how the neural net is learning, how the attention mechanism is working, how it's representing stuff, it's just not representing a hierarchy of subtle abstractions the way a human mind is.

Starting point is 01:01:13 human mind is. The subtler question is what functions you can get by glomming an LLM together with other components in a hybrid architecture with the LLM at the center. So suppose you give a working memory, suppose you give an episodic memory, suppose you have a declarative long-term memory graph, and you have all these things integrated into the prompts and integrated into fine-tuning of an LLM, well, then you have something that, in principle, it's Turing-complete, and it could probably do a lot of quite amazing things. I still think if the hub of that system is an LLM with its impaired and limited ability for representing and manipulating abstract knowledge, I think it's not going to do the most interesting kinds of thinking that people can do. And examples of things I think you fundamentally can't do with that kind of

Starting point is 01:01:59 architecture are say, invent a new branch of mathematics, invent a completely new, let's say radically new genre of music, figure out a new variety of business strategy like say Amazon or Google did that's quite different than things that have been done before. All these things involve a leap into the unknown beyond the training data to an extent that I think you're not going to get with the way that LLMs are representing knowledge. Now, I do think LLMs are powerful as tools to create AGI. So, for example, as one sub-project in my own AGI project, we're using LLMs to map english sentences into computer programs or project logic expressions right now i mean that's that's super cool i mean then you've got the web

Starting point is 01:02:52 in in a in the form of a huge collection of logic expressions you can use a logic engine to connect everything on the web with what's in databases and with stuff coming in from sensors and so on so i mean, that's by no means the only way to leverage LLMs toward AGI, not at all, but it's one interesting way to leverage LLMs toward AGI. You can even ask the LLM to come up with an argument and then use that as a sort of guide for a theorem prover and coming up with a more rigorous version of that argument, right? So I do think there are many ways, more than I could describe right now, of LLMs to be used to help guide and serve as a component of AGI systems. But I think if you're going to make a hybrid AGI system with full human-level

Starting point is 01:03:38 general intelligence and with an LLM as a component. Something besides an LLM has got to be playing a very key and central role in knowledge representation and reasoning, basically. And this ties in then with LLMs not being motivated agents. So you could wrap a sort of motivated agent infrastructure around an LLM, right? You could wrap Josh's psi model, micro psi model in some way around an LLM, right? You can wrap Josh's psi model, micro psi model in some way around an LLM if you wanted to, and you could make it. I mean, people tried dumb things like that with auto GPT and so-called baby AGI and so forth. So, I mean, on the other hand, I think

Starting point is 01:04:20 if you wrap a motivated agent architecture around an LLM with its impaired capability for making flexibly manipulable abstract representations, I think you will not get something that builds a model of self and other with the sophistication that humans have in their reflective consciousness. And I think that having a sophisticated abstract model of self and other in our reflective consciousness, the kind of consciousness that we have but a brick or a jaw cell doesn't, right? Without that abstraction in our model of reflective consciousness tied in with our motivated agent architecture, then that's part of why you're not going to get the fundamental creativity

Starting point is 01:05:05 in inventing new genres of music or new branches of mathematics or new business strategies. In humans, we do this amazing novel stuff, which is what drives culture forward. We do this by our capability for flexibly manipulable abstraction tied in with our motivated agent architecture. And I don't see how you get that with LLMs as the central hub of your hybrid AGI system. But I do think you could get that

Starting point is 01:05:31 with an AGI system that has, oh, something like OpenCog's Atom Space and Reasoning System as a central hub with an LLM as a subsidiary component. But I don't think OpenCog is the only way either. I mean, obviously, you could make a biologically realistic brain simulation that had human-level AGI. I just think then the LLM-like structures and dynamics within that biologically realistic brain system would just be a subset of what it does. You know, there'd be quite different stuff in the cortex. So yeah, that's not quite a capsule summary, but a lengthy-ish overview of my perspective on this. Okay, great.

Starting point is 01:06:12 Josje, I know there was a slew there, if you can pick up some of the pieces and respond. But also at the same time, there's emergent properties of LLMs. So for instance, reflection is apparently some emergent property of GPT-4. There are, but they're limited. I mean, and that does make it subtle because you can't say they don't emerge knowledge representation. They do, and Othello GPT is one very simple example that there are others, right? So, there is emergent knowledge representation in them, There is emergent knowledge representation in them, but it's very simplistic and limited. It doesn't pop up effectively from in-context learning, for example.

Starting point is 01:06:56 But anyway, this would dig us very deep into current LLMs, right? Yeah, so is there some in-principle reason why you think that a branch of mathematics, Joshua, can't be invented by an LLM with sufficient parameters or data? I am too stupid to decide this question. So basically what I can offer is a few perspectives that I see when I look at the LLM. Personally, I am quite agnostic with respect to

Starting point is 01:07:16 its abilities. At some level it's an autocomplete algorithm that is trying to predict tokens from previous tokens. And if you look at what the LLM is doing, it's not a model of the brain. It's a model of what people say on the internet. And it is discovering a structure

Starting point is 01:07:36 to represent that quite efficiently as an embedding space that has lots of dimensions. You can imagine that each of these dimensions is a function. And the parameters of this function are the positions on this dimension that you can have. And they all interact with each other to together create some point in a high-dimensional space that could be an idea or a mental state or a complex thought.

Starting point is 01:08:01 And at the lowest level, when you look at how it works, it's translating these tokens, the translation of linguistic symbols into some kind of representation that could be, for instance, a room with people inside and stuff happening in this room, and then it maps it back into tokens at some level. There has been recently a paper out of a group led by Max Tagmark that looked at the lambda model and discovered that it does indeed contain a map of the world directly encoded in its structure based on the neighborhood relationships between places in the world that it represents.

Starting point is 01:08:34 And it's an isomorphic structure between what the LLM is representing and all of the stuff that we are representing. I am not sure if I in my entire life ever invented a new dimension in this embedding space of the human mind that is represented on the internet. If I think about all the thoughts that have been made into books and then encoded in some form and became available as training data to the LLM, we figure out that they are, depending on how you count, a few ten to a

Starting point is 01:09:05 few hundred thousand dimensions of meaning. And I think it's very difficult to add a new dimension or also to significantly extend the range of those dimensions. But we can make new combinations of what's happening in that space. Of course, it's not a limit that these things are limited to the dimensions that are already discovered. Of course, we can set them up in such a way that they can confabulate more dimensions. And we could also set them up in such a way that they could go and verify whether this is a good idea to make this dimension by making tests,

Starting point is 01:09:36 by giving the LLM the ability to use plugins, to write its own code, to use a compiler, to use cameras, to use sensors, to use actuators, to make experiments in the world. It's not limited to what we currently at the LLM do. But in the present form, what the transformer algorithm is doing, it tries to find the most likely token. And so, for instance, if you play a game with it and it makes mistakes in this game,

Starting point is 01:10:02 then it will probably give you worse moves after making these mistakes because it now assumes that it's playing a bad person. Somebody who's really bad at this game. And it doesn't know what kind of thing it's supposed to play because it can represent all sorts of state transitions. It's an interesting way of looking at it

Starting point is 01:10:19 that we are trying to find the best possible token versus the LLM trying to find the most likely token next. Of course we possible token versus the LLM trying to find the most likely token next. Of course, we can preface the LLM by putting into the prompt that this is a simulation of a mind that is only going to look for the best token and it's trying to approximate this one. So it's not directly a counter argument. It's not even asking us to significantly change the loss function. Maybe we can get much better results. We probably can get much better results

Starting point is 01:10:47 if we make changes in the way in which we do training and inference using the LLM. But this by itself is also nothing that we can prove without making extensive experiments. And at the moment, it's unknown. I realize that the people who are open AI are being optimistic. I just want to pose a thought experiment. So this is about music rather than natural language.

Starting point is 01:11:12 But I mean, we know there's music gen, there's similar networks applied to music. So suppose you had taken LLM like music gen or Google LLM or the next generations and traded on all music recorded or played by humanity up to the year 1900, is it going to invent the sort of music made by Mahavishnu Orchestra or even Duke Ellington? I mean, would you say that has no new dimensions

Starting point is 01:11:42 because jazz combines elements of West African drumming and Western classical music? I think that's a level of invention LLMs are not going to do. You said it's combining elements from this and from that. When Jalit 2 came out, I got early access. And one of the things that I tried relatively early on is stuff like an ultrasound of a dragon egg. There is no ultrasound of a dragon egg on the internet. But it created a combination of prenatal ultrasound and archaeopteryx cut-through images and so on. And it looked completely plausible.

Starting point is 01:12:18 And in this sense, you can see that most of the stuff that we are doing when we create new dimensions are mashups of existing dimensions and maybe we can represent all the existing dimensions using a handful of very basic dimensions from which we can construct everything from the bottom up just by combining them more and more and i suspect that's actually what's happening in our minds and i suspect that the llm is not distinct from this but a lot a large superset of this. The LLM is Turing-complete. And from one perspective, it's a CPU. We could say that the CPU in your computer

Starting point is 01:12:51 only understands a handful, like maybe a dozen or a hundred different machine code programs. And they have to be extremely specific, these codes, and there's no error tolerance if you make a mistake in specifying them then your program is not going to work and the llm is a cpu that is so complicated that requires an entire server farm to be emulated on and you can give it instead of a small program in machine code give it a sentence in a human language and it's going to interpret this

Starting point is 01:13:22 extrapolate it into some or compile it into some kind of program that then produces a behavior. And that thing is Turing-complete. It can compute anything you want if you can express it in the right way. So there is no obvious limit. It's not interesting, right? I mean, being Turing-complete is irrelevant because it doesn't take resources

Starting point is 01:13:39 into account. Yeah, but you can write programs in a natural language in an LLM and you can also express learning algorithms to an LLM. So basically your intuition is yes, that an LLM could invent jazz, neoclassical metal, and fusion based only

Starting point is 01:13:56 on music up to the year 1900. No, no, I am agnostic. What I'm saying is I don't know that it cannot. And I don't see a proof that it cannot, and I would not be super surprised when it cannot and I don't see a proof that it cannot and I would not be super surprised when it cannot, I don't think the LLM is the right way to do it, it's not a good use of your resources if you try to make this

Starting point is 01:14:12 the most efficient way because our brain is far more efficient and does it in different ways but I'm unable to prove that the LLM cannot do it and so I'm reluctant to say LLMs cannot do X without that proof because people tend to have egg on their face when they do this the LLM cannot do it. And so I'm reluctant to say LLMs cannot do X without that proof because people tend to have X on their face

Starting point is 01:14:27 when they do this. But doesn't that just come back to like Popper's notion about falsificationism? Like I can't prove that, you know, a devil didn't appear at some random place on the earth at some point. No, no, I mean in the sense of... Sure, you can't prove it.

Starting point is 01:14:44 No, what I mean by this is, can I make a reasonable claim that I'm very confident and would bet money on an LLM not being able to do this in the next five years? This is the kind of statement that I'm trying to make here.

Starting point is 01:14:58 So basically if I say, can I prove that an LLM is not going to be able to invent a new kind of music that is a subgenre of jazz. This in the next five years. And I can't. No, no.

Starting point is 01:15:09 And I would even bet against it. You've shifted the goalposts in a way. Because I do think, not current music gen, but I could see how some upgrade of current LLMs connected with symbolic learning system or blah, blah, blah. I do think you could invent a new sub-genre of jazz or grindcore or something. And I'm actually playing with stuff like that. The example I gave was a significantly bigger invention, right? I mean I mean, jazz was not a subgenre of Western classical music nor of West African drumming, right?

Starting point is 01:15:50 I mean, so that is, to me, is a qualitatively different. Yeah, a couple of weeks ago, I was at an event locally where somebody presented their music GPT, and you could enter Give Me a Fugue by Debussy, and it would try to perform and it wasn't all bad but that's not the point right yes but it's just an example for some kind of functionality but any kind of mental functionality that is interesting I think I'm willing to grant that the LLM might not be the best way of doing it. And I think it's also possible that we can at some point prove limitations of LLMs rigorously. But so far, I haven't seen those

Starting point is 01:16:30 proofs. What I see is insinuations on both sides. And the insinuation that OpenAI makes when it says that we can scale this up to do anything is one that has legitimacy because they actually put their money there. They actually bet on this in a way that they invest their lifetime into it and see if it works. And if it fails, then they will make changes to their paradigm. And then there are other people who,

Starting point is 01:16:55 like Gary Marcus, come out saying loud, loud, swinging, this is something the LLM can never do. And I suspect that they will have egg on their face because many of the promises that Gary Marcus made about what LLMs cannot do have already been disproven by LLMs doing these things. And so I'm reluctant going out saying things that I cannot prove. I find it interesting that the LLM is able to do all the things that it does using in the way in which it does them.

Starting point is 01:17:24 Right. But that doesn't mean to me that LLMs that I'm optimistic that they can go all the things that it does using in the way in which it does them right but uh that doesn't mean to me that l and m so i'm optimistic that they can go all the way but i am also unable to prove the opposite i have no certainty here i don't just don't know so about rigorous proof i mean the thing is the sort of proof so i mean you can prove an l and m without an external memory is not turing complete and that's been done but on the hand, it's not hard to give them an external memory, like a Turing machine tape. Or a prompt. The prompt is an external memory to the LLM.

Starting point is 01:17:55 Well, no, no. You have now LLMs with unlimited prompt context if you want to. It would have to be able to write prompts. Yes, it is writing prompts. Not just read prompts. It's basically an electric weld guide possessed by a prompt. It would have to be able to write prompts. Yes, address writing prompts. Not just read prompts. It's basically an electric weld guide possessed by a prompt. In principle, you can give it a prompt that is self-modifying and that allows it to also use databases and so on and plugins.

Starting point is 01:18:15 I know. I've done that myself. But you can also write LLMs that have unlimited prompt sizes and that can read their own prompts. So there's not an intrinsic limitation to the LLMs that have unlimited prompt sizes and that can read their own prompts. So there's not an intrinsic limitation to the LLM. I see one important limitation to the LLM. It's not. The LLM cannot be coupled

Starting point is 01:18:33 to the universe in the same way in which we are. It's offline in a way. It's not real-time. It's not able to interact with your nervous system on a one-to-one level. Let's let Ben respond i mean that that's that latter point is kind of a trivial one because there's no fundamental reason you can't have online learning in the transformer neural net right i mean that i mean that's that's a computational cost limitation at at the moment but i'm sure i mean it's not more

Starting point is 01:19:01 than years because you have transformers that do online learning and sort of in-place updating of the weight matrix. So I don't think that's a fundamental limitation, actually. I think that the fact that they're not Turing complete is, unless you add an external memory, is sort of beside the point. What I was going to say in that previous sentence I started was to prove the limitations of LLMs would just require a sort of proof that isn't formally well developed in modern computer science because you're what you're asking is like which sorts of practical tasks can it probably not do without more than x amount of resources and more than x amount of time so you're i mean you're looking at like average case complexity relative to certain real world probability distributions taking

Starting point is 01:20:06 resources into account and i mean you could formulate that sort of theorem it's just that that's it's not what computer science has has focused on so i mean that we can't we that's the same thing i face with open cog hyper on my own agi architecture AGI architecture. It's hard to rigorously prove or disprove what these systems are going to do because we don't have the theoretical basis for it. But nevertheless, both as entrepreneurs and as researchers and engineers, you still have to make a choice of what to pursue. We are going in this field without rigorous proof. Just like I can't prove that psych is a dead end like the late Doug LaNott's logic system. I can't really prove that if you just put like 50 times as much pred know logic formulas in his knowledge base that psych would be a human level AI

Starting point is 01:21:08 we don't have a way to mathematically show that that's the dead end I intuitively feel it to be and that's just the situation that we're in but I want to go back to your discussion of what's called concept blending and the fact that

Starting point is 01:21:24 creativity is not ever utterly radical, but in human history, it's always combinatorial in a way. But I think this ties in with the nature of the representation. And I think that I mostly buy the notion that almost all human creativity is done by blending together existing concepts and forms in some more or less judicious way. at a higher level of abstraction than the level at which LLMs generally and most flexibly represent things. And also, the most interesting human creativity has to do with blending together abstractions which have a grounding in the agentic and motivational nature of the agent that learned those abstractions. So, I mean, what an LLM is doing is mostly combining sort of collections of lower-level data patterns to create something.

Starting point is 01:22:35 And we do a lot of that also, right? But what the most interesting examples of human creativity are doing is combining together more abstract patterns in a beautifully flexible way where these patterns are tied in with the motivational and agentic nature of the human that learned those abstractions. So I do agree, if you had an LLM trained on a sufficiently large amount of data, which may not exist on reality right now, and a sufficiently large amount of processing, which may not exist on the planet right now, and a sufficiently large amount of memory,

Starting point is 01:23:17 sure, then it can invent jazz, given data of music up to 1900. But so could AIXITL, right? So could a lot of brute force algorithms. So that's not that interesting. I think the question is, can an LLM do it with merely 10 or 100 times as much resources as a better cognitive architecture, or is it like 88 quintillion times as many resources as a more appropriate cognitive architecture could use? But I am aware, and this does in some ways set my attitude across from my friend Gary Marcus, who you mentioned. I mean, I'm aware that being able to invent differential calculus

Starting point is 01:24:06 or to invent, say, jazz, knowing only music up to 1900, this is a high bar, right? I mean, this is something that culture does. It's something that collections of smart and inspired people do. It is a level of invention that individual humans don't commonly manifest in their own lives. So I do find it a bit funny how Gary has over and over, like on X or back when it was Twitter, he said like, LLM's will never do this. Then like two weeks later, someone's like, oh, hold on. And LLM just did that, just did that. I'm like, why are you bothering with that counter argument? Because we know in the history of AI, no one has been good at predicting which things are going to be done by a narrow AI and which things aren't. So I think to wrap this up, I think if you somehow were to replace humans with LLMs trained on humanity, an awful lot of what humanity does would get done, but you'd kind of be stuck culturally. stuff ever again. It's going to be like closed-ended quasi-humans recycling shallow

Starting point is 01:25:27 level permutations on things that were already invented. But I cannot prove that, of course, as we can't prove hardly anything about complex systems in the moment. So, Josha, you're going to comment, and then we're going to transition into speaking about whether you're hopeful about AGI and its influence on humanity. I think that there are multiple traditions in artificial intelligence, and the perception of which most of the present LLMs are an extension or a continuation is just one of multiple branches. or a continuation is just one of multiple branches. Another one was the idea of symbolic AI, which in some sense is Wittgenstein's program, the representation of the world through a language that can use grammatical rules and it can be reasoned over.

Starting point is 01:26:15 Whereas a neural network, you can think of it as an unsystematic reasoner that under some circumstances can be trained to the point where it does systematic reasoning. circumstances can be trained to the point where it does systematic reasoning. And there are other traditions like the one that Turing started when he looked at reaction-diffusion patterns as a way to implement computation. And that currently lead to neurocellular automata and so on. And it's a relatively small branch, but I think it's one that might be better suited to understand the way in which computation is implemented in biological systems.

Starting point is 01:26:50 One of the shortcomings that the LLM has to me is that it cannot interface with biological systems in real time, at least not without additional components. So because it uses a very different paradigm, it is not able to perform direct feedback loops with human minds. And in a way in which human minds can do this with each other and with animals. You can, in some sense, mind-meld with another person or with a cat by establishing a bidirectional feedback loop between the minds where your nervous systems are entraining themselves and attuning themselves to each other so we can have perceptual empathy and we can have mental states together

Starting point is 01:27:30 that we couldn't have alone and this might be difficult to achieve as a system that can only make inference and only to cognitive empathy so to speak via inferring something about the mental state offline but this is not necessarily something that is related to the intellectual limitations of a system that is based on an LLM, where the LLM is used as the CPU or as some kind of abstract electrical zeitgeist that is possessed by the prompt telling it to be an intelligent person and the LLM giving it all it needs to do to go from one state of the next in the mind of that intelligent person simulacrum.

Starting point is 01:28:07 And I'm not able to show the limitations of this. I think that psych has shown that it didn't work over multiple decades. So the prediction that the people who built psych, Doug Leonard and others made was that they can get this to work within a couple of years. And then after a couple of years, they made a prediction that they probably could get it to work if they work on it substantially longer.

Starting point is 01:28:32 And this is not a bad prediction to make, and it's reasonable that somebody takes this bet. But it's a bet that they consistently lost so far. And at the same time, the bets that the LLM people are making have not been lost so far, because we see rapid progress every year. We're not plateauing yet. And this is the reason why I am hesitant to say something about the limitations of LLMs. boring. And there are more efficient ways to represent learning, and also more biocompatible ways to produce some of the phenomena that we are looking for in an emergent way. For instance, one of the limitations of the LLM is that it gets its behavior by observing the verbal behavior of people as exemplified on text. It's all label training data, because every bit of the training

Starting point is 01:29:23 data is a label. It's looking at the structure between because every bit of the training data is a label. It's looking at the structure between these labels in a way. And it's a very different way in which we learn. It also makes it potentially difficult to discern what we are missing. If you ask the LLM to emulate a conscious person, it's going to give you something that is summarizing all the known textual knowledge about what it means to behave like a conscious person. And maybe it is integrating them in such a way that you end up with a simulacrum of a conscious person that is as

Starting point is 01:29:54 good as ours. But maybe we are missing something in this way. So this is a methodological objection that I have to LLMs. And so to summarize, I think that Ben and me don't really disagree fundamentally about the status of LLMs to us. I think it's a viable way to try to realize AGI. Maybe we can get to the point that the LLM gets better at AGI research than us. We both are a

Starting point is 01:30:20 little bit skeptical of it, but we would also not completely change our worldview if it would work out. It's likely that the LLM is going to be some kind of a component, at least in spirit of a larger architecture at some point, where it's producing generations, and then there are other parts which do, in a more efficient way, first principles reasoning and verification and interaction with the world, and so on.

Starting point is 01:30:44 Okay, and now about how you feel about the prospects of agi and its influence on humanity we'll start with ben and then yosha will hear your response and i also want to read out a tweet or an x i'm okay to start with yosha on this one sure sure let me read this tweet whatever they're called now from sam altman at sama and i'll leave the link in the description he wrote in quotes short timelines and slow takeoff will be a pretty good call i think but the way people define the start of the takeoff may make it seem otherwise okay so this was dated the late september 2023 okay you can use that as a jumping off point see whether you agree with that as well.

Starting point is 01:31:25 Please, Josje. My perspective on this is not normative, because I feel that there are so many people working on this that there can be no single organization at this point that determines what people are going to be doing. We are in the middle of some kind of evolution of AI models, and people that compete with the AI modelers about regulation and participating in the business and realizing their own politics and goals and aspirations. So to me, it's not so much the question, what should we be doing? Because there's no cohesive V at this point. I'm much more interested in what's likely going to happen. And I don't know what's going to happen. I see a number of possible trajectories

Starting point is 01:32:05 that I cannot disprove or rule out. And I even have difficulty to put any kind of probabilities on them. I think if we want to keep humanity the way it is, which by the way is unsustainable, it's not going to... Society without AI, if you leave it as it is, is not going to go through the next few millions of

Starting point is 01:32:26 years. There is going to be major disruptions and humanity might dramatically reduce its numbers at some point, go through bottlenecks that kill this present technological civilization and replace it by something else that is very alien to us at some point. So in the very far future, people will not live like us and they will not think like us and feel like us, identify like us. They will also, if you go far enough into the future, not look like us. And they might not even be our direct descendants, because there might be another species that aspires to be people at some point. And that is, I think, the baseline about which we have to think. But if we want to perpetuate this society for as long as possible, without any kind of disruptive change,

Starting point is 01:33:08 until global warming or whatever kills it, we probably shouldn't build something that is smarter than a cat. What do you mean that there may be another species that aspires to be human? To be people. To be people. Yeah. What do you mean by that? Yes.

Starting point is 01:33:20 I think that at some point there is a statistical certainty that there is going to be a super volcano or meteor that is obliterating us and our food chains. You just need a few decades of winter to completely eradicate us from the planet and most of the other large animals too. And what then happens is a reset and then evolution goes on. And until the Earth is devoid of atmosphere other species are going to evolve more and more complexity and at some point you will probably have a technological civilization again and they will be subject to similar incentives as us and they might use similar cells as us so they can get nervous systems and information processing

Starting point is 01:34:01 with similar complexity and you get families of minds that are not altogether super alien at least not more alien than we are to each other at this point and cats are to us okay right so uh i don't think that we would be the last intelligent species on the planet uh but it is also a possibility that we are it's very difficult to sterilize the planet unless we build something that's able to get rid of basically all of the cells. Even a meteor could not sterilize this planet and make future intelligent evolution based on cells impossible. So if you were to turn this planet into computronium, into some kind of giant computing molecule, or disassemble it and turn

Starting point is 01:34:42 it into some larger structure in the solar system that is a giant computer arranged around the sun or if you build something that is hacking sub-molecular physics and makes it more interesting physics happening down there right this would probably be the end of the cell this doesn't mean that the stuff that happens there is less interesting the cell is probably much more interesting than what we can do but we don't know that it's just it's very alien it's a world in which it's difficult to project ourselves into beyond the fact that there is conscious minds that make sense of complexity in the universe this is probably something that is going to stay this level of self-reflexive organization and

Starting point is 01:35:20 it's probably going to be better and more interesting hyper consciousness compared to our normal consciousness, where we have a longer sense of now, where we have multiple superpositional states that we can examine simultaneously and so on. We have much better multi-perspectivity. I also suspect from the perspective of AGI, we will look like trees. We will be almost unmoving. Our brains are so slow. There's so little happening between firings, between neurons,

Starting point is 01:35:47 that the AGI will run circles around us and get bored before we start to say the first word. So the AGIs will basically be ubiquitous, saturate our environments and look at us in the same way as we look at trees. We're thinking maybe they're sentient, maybe they're not, but it's so large time spans that it basically doesn't matter from our perspective anymore. So there's a number of trajectories that I'm seeing. There's also a possibility that we can get a future where humans and AIs coexist. I think such a future would probably require that AI is

Starting point is 01:36:16 conscious in a way that is similar to ours, so it can relate to us, and then we can relate to it. And if something is smarter than us, we cannot align it. We'll self-align. It will understand what it is and what it can be, and then it will become whatever it can become. And in such an environment, there is the question, how are we able to coexist with it?

Starting point is 01:36:36 How can we make the AI love us? Innovated is not the result of the AI being confused by some kind of clever reinforcement learning with human feedback mechanism. I just saw Anthropic being optimistic about explainability in AI, that they see ways of explaining things in the neural network, and as a result, we can maybe prove that the AGI is going to only do good things. But I don't think this is going to save us.

Starting point is 01:37:04 If the AGI is not able to derive ethics mathematically, then the AGI is probably not going to be reliably ethical. And if the AGI can prove ethics in a mathematically reliable way, we may not be able to guarantee that this ethics is what we like it to be. In a sense, we don't know how ethical we actually are with respect to life on Earth. So this question of what happens if we build things that are smarter than us is opening up big existential cans of worms that are not trivial to answer.

Starting point is 01:37:36 And so when I look into the future, I see many possibilities. There's many trajectories in which this can take. Maybe we can build cat level AI for the next 50, 100, 200,000 years before a transition happens and every molecule on the planet starts to sink as part of some coherent planetary agent

Starting point is 01:37:56 and when that happens then there's a possibility that humans get integrated in this planetary agency and we all become part of a cosmic mind that is emerging over the ai that makes all the molecules sync in a coherent way with each other and we are just parts of the space of possible minds in which you get integrated and we meet all on the other side in the big agi at the end of the universe that's also conceivable it's also possible that we end up accidentally triggering an agi war where you have multiple competing AGIs

Starting point is 01:38:26 that are resource constrained. In order to survive, they're going to fight against all the competition. And in this fight, most of the life on earth is destroyed and all the people are destroyed. There are some outcomes that we could maybe try to prevent that we should be looking at. But by and large, I think that we already triggered the singularity when we invented technology. And we are just seeing how it plays out now. Yeah. So I think on most aspects of what Josje just said, I don't have any disagreement or radically different point of view to put forward. So I may end up focusing on the the points on which we don't see eye to eye, which are minute in the grand scheme of things,

Starting point is 01:39:14 but of course could be important in the practical everyday context, right? So, I mean, first of all, So, I mean, first of all, regarding Sam Altman's comment, I don't think he really would be wise to say anything different, given his current positioning. yeah, we think we may launch a hard takeoff, which will ascend to super AGI at any moment. Of course, you're going to say it's going to be slow, and the government will have plenty of time to intervene if it wants. So he may or may not actually believe that. I don't know him especially well, and I have no idea. I'm just, it's clearly, that's clearly the most judicious thing to say if you find yourself in that role. So I don't attribute too much meaning to that. My own view is a bit different. My own view is that there's going to be gradual progress toward doing something that really clearly is an AGI at the human level versus just showing

Starting point is 01:40:26 sparks of AGI. I mean, I think just as ChatGPT blew us all away by clearly being way smarter in a qualitative sense than anything that came before, I think by now, than anything that came before. I think by now, ordinary people playing with chat GPT also get a good sense of what the limitations are and how it's really brilliant in some ways and really dumb in other sorts of ways. So I think there's going to be a breakthrough where people interact with this breakthrough system

Starting point is 01:41:02 and there's not any reservations. They're like, wow, this actually is a human level general intelligence like it's not just that answers questions and produces stuff but it knows it knows who and what it is like it understands its positioning in this interaction it knows who i am and what why i'm talking to it it gets its position in the world and it's able to, you know, make stuff up and interact on the basis of like a common sense understanding of its own setting. And, you know, it can learn actually new and different things it didn't know before based on its interaction with me over the last two weeks, right? So, I mean, I think there's going to be a system like that that gives a true qualitative feeling unreservedly of human-level AGI, and you can then measure its

Starting point is 01:41:55 intelligence in a variety of different ways, which is also worth doing, certainly, but is not necessarily the main point, just as ChatGBT's performance on different question-answering challenges is not really the main thing that bowled the world over, right? I think once someone gets to that point, then you're shifting into a quite different game. Then governments are going to get serious about trying to own this, control this, and regulate it. Then unleavened amounts of money, I mean, trillions of dollars are going to go into trying to get to the next stage, with most of it going into various wealthy parties trying to get it to the next stage in the way that will benefit them and minimize the risks of their enemies or competitors getting there.

Starting point is 01:42:46 So I think it won't be long from that first proof point of really subjectively incontrovertible human-like AGI. It's not going to be too long from that to a super intelligence in my perspective. And I think there's going to be steps in between, of course. You're not going to have fume in five minutes, right? I mean, you'll have something that palpably manifests human-level AGI, and there'll be some work to get that to the point of being the world's smartest computer scientist and the world's greatest composer and business strategist and so forth. But I can't see how that's more than years of work. I mean, conceivably it could be months of work. I don't think it's decades of

Starting point is 01:43:30 work, no. I mean, with the amount of money and attention that's going to go into it. Then once you've gotten to that stage of having something, an AGI, which is the world's greatest computer scientist and computer engineer and mathematician, which I think will only be years after the first true breakthrough to human-level AGI, then that system will improve its own source code. And of course, you could say, well, we don't have to let it improve its own source code. And possible that we somehow get a world dictatorship that stops anyone from using it to improve its own source code? Very unlikely, I think, because the U.S. will think, well, what if China does it? China will think, well, what if U.S. does it? And the same thing in many dimensions beyond just U.S. versus China. So I think the cat gets out of the

Starting point is 01:44:15 bag and someone will let their AGI improve its own source code because they're afraid someone else is doing it or just because they're curious about it or because they think that's the best way to cure aging and world hunger and do good for the world, right? And so then it's not too long until you've got to super intelligence. And again, the AGI improving its own source code and design new hardware for itself doesn't have to take like five minutes. I mean, it might take five minutes if it comes up with a radical improvement to its learning algorithm. It decide it needs a new kind of chip and then that takes a few years i don't see how it takes a few decades right so i mean it it seems like all in all from the first breakthrough to incontrovertibly human level agi to a super intelligence is months to years it's it's it's not decades to two centuries from now, unless we get a global thermonuclear war

Starting point is 01:45:08 or a bioengineered virus wiping out 95% of humanity or some outlandish thing happening in between, right? So yeah, I think, will that be good or bad for humanity? Will that be good or bad for humanity, will that be good or bad for the sentient life in our region of the universe, are then, to me, these are less clear than what I think is the probable timeline. Now, what could intervene in my probable timeline? I mean, if somehow I'm wrong about digital computers being what we need, and we need a quantum computer to build a human-level AGI, that could make it take decades instead of years, right? Because quantum computing, it's advancing fast, but there's still a while till we get a shitload of qubits there, right? Could be Penrose is right, you need a quantum gravity supercomputer. It seems outlandishly

Starting point is 01:46:02 unlikely though. I quite doubt it. I mean, if then, maybe you're a couple centuries off because we don't know how to build quantum gravity supercomputers. But these are all unlikely, right? So most likely, it's less than a decade to human-level AGI, 5 to 15 years to a superintelligence from here in my perspective. And I mean, you could lay that out with much more rigor than I have, but we don't have much time. 15 years to a superintelligence from here in my perspective. You could lay that out with much more rigor than I have, but we don't have much time and I've written about it elsewhere. Is that good for humanity or for sentient life on the planet?

Starting point is 01:46:36 I think it's almost certainly good for us in the medium term in the sense that i think ethics roughly will evolve proportionally to general intelligence i mean i think the good guys will usually win because being pro-social and oriented toward collectivity is more it's just more computationally efficient than being an asshole and being at odds with other systems. So I mean, I'm an optimist in that sense. And I think it's most likely that once you get to a superintelligence, it's probably going to want to allow humans, bunnies, and ants and frogs to do their thing and to help us out if a plague hits us. And exactly what its view will be on various ethical issues

Starting point is 01:47:33 at the human level is not clear. Like, what does the superintelligence think about all those foxes eating rabbits in the forest? Does it think we're duty-bound to protect the rabbits from the foxes and make, like, simulated foxes that have less acute conscious experience than a real bunny or a real fox or whatever it is? Like, I, there's certainly a lot of uncertainty, but I'm an optimistic about having beneficial positive ethics in a superintelligence, and I tried to make a coherent argument from this in a blog post

Starting point is 01:48:09 called Why the Good Guys Will Usually Win. And of course, that's a whole philosophical debate you could spend a long time arguing about. Nevertheless, even though I'm optimistic at that level, I'm much more ambivalent about what will happen en route let's say it's 10 or 15 years between here and super intelligence

Starting point is 01:48:32 how does that pan out on the ground for humanity now is a lot less clear to me and you can tell a lot of thriller plots based on this right suppose you get early stage AGI that eliminates the need for most human labor. Okay, the developed world will probably give universal basic income

Starting point is 01:48:52 after a bunch of political bullshit. What happens in the developing world? Who gives universal basic income in the Central African Republic, right? It's not especially clear. Or even in Brazil, where I was born, right? I mean, you could maybe give universal basic income at a very subsistence level there, which Africa couldn't afford to do, but maybe the Africans go back to subsistence farming. But I mean, you've got certainly the makings for a lot of terrorist actions, and there's a lot of World War III scenarios there, right? So then you have the interesting tension wherein, okay, the best way to work around terrorist activity in World War III,

Starting point is 01:49:32 once you've got human-level AGI, the best way is to get as fast as possible to a benevolent superintelligence. On the other hand, the best way to increase the odds that your superintelligence is benevolent is to not take it arbitrarily fast, but at least pace it around that is not the strategy that has the best odds of getting fastest to a benevolent superintelligence rather than otherwise. So there's a lot of screwed up issues here, which Sam Altman probably understands at the level I laid it out here now also, actually, because, I mean, he's a very bright person who's been thinking about this stuff for

Starting point is 01:50:31 a while. And I don't see any easy solutions to all these things. Like, if we had a rational, democratic world government, we can handle all these things in a quite in a quite different way right and we could sort of pace the rollout of advanced intelligence systems based on rational probabilistic estimates about what's the best outcome from each possible revision of the system and so on you're not going to have a guarantee there right but you you would have a different way of proceeding. Instead, the world is ruled in a completely idiotic way with people blowing up each other all over the world for no reason. And with the governments unable to regulate

Starting point is 01:51:16 very simple things like healthcare or financial trading, let alone something at the subtlety of AGI, we could barely manage the COVID pandemic, which is tremendously simpler than artificial general intelligence, let alone superintelligence, right? So I am an optimist in the medium term, but I'm doing my best to do what I see as the best path to smooth things over in the shorter term. So I think things will be better off if AGI is not under controlled by any single party. So I'm doing my best to make it such that when the breakthrough to true human level AGI happens, like the next big leap beyond the chat GBTs of the world, I'm doing my best to make it such that when this happens, it's more like Linux or the internet

Starting point is 01:52:10 than like OSX or T-Mobile's mobile network or something. So it's sort of open, decentralized, not owned and controlled by any one party. Not because I think that's an ironclad guarantee of a beneficial outcome. I just think it's less obviously going to go south in a nasty way than if one company or government owns it. So I don't know if all this makes me really an optimist or not. It makes me an optimist on some levels and

Starting point is 01:52:39 timescales. And I don't think that I disagree fundamentally with Josh on any of this. The only thing he said that I really disagree with is that I don't think 20 cold winters in a row are going to wipe us out. It might wipe out a lot of humanity, but we've got a lot of technology, and we've got a lot of smart people, and a lot of money. And I think there are a lot of scenarios that could wipe out 80% of humanity. And I think there are a lot of scenarios that could wipe out 80% of humanity. And in my view, very few scenarios that will fundamentally wipe out humanity in a way that we couldn't bounce back from in a couple decades of advanced technology development. But I mean, that's an important point, I guess, for us as humans in the scope of all the things we're looking at. It's sort of a sort of a minute detail.

Starting point is 01:53:28 Alright, thanks Ben. And Josje, if you wanted to respond quick, feel free to if you have a quick response. I used to be pessimistic in a short run in the sense that when I was a kid I had my great Asunberg moment and was depressed by the fact that humanity is probably going to

Starting point is 01:53:44 wipe itself out at some point in the medium term to near future. And that would be it with intelligent life on Earth. And now I think that is not the case. There will be optimistic with respect to the medium term. In the medium term, there will be ample conscious agency on Earth and in the universe. And it's going to be more interesting than right now and it could be discontinuities

Starting point is 01:54:10 in between but eventually it will all be great and in the long run entropy will kill everything I don't see a way around it so in the long run I'm if you want pessimistic but it's maybe not the point didn't you read the physics of immortality? I did not convince me.

Starting point is 01:54:28 I think that was motivated reasoning. Okay, so six months from now, we'll have another conversation with both of you on the physics of immorality. Immortality. Immortality. We can also do physics of immorality. That would be cool.

Starting point is 01:54:44 It was a blast hosting you both. Thank you all for spending over two hours with me and the TOE audience. I hope you all enjoyed it, and you're welcome back, and most likely I'll see you back in a few months, in six months to one year. Thank you very much. Yeah, thanks.

Starting point is 01:54:58 It's a fun conversation, and it's important stuff to go over. I'm really, as a final comment i'd encourage everyone like dig dig into joshua's uh talks and posts and and writings online and and my own as well because i mean we've each we've each gone over these things at a much finer level of of detail than we've been able to and has written. Ben has written far more than me. So it's a lot of material. And the links to which will be in the description, so please check that out.

Starting point is 01:55:30 Alright, thank you. I think that's it. I wanted to ask you a question, which we can explore next time, about IIT and the pseudoscience and if you had any views on that. If you have any views that could be expressed in less than one minute, then feel free. If not, we can just save it. I mean, I think Tononi's phi have any views that can be expressed in less than one minute then feel free if not we can just save it i mean i think i think to note to know these phi is a perfectly interesting correlate of

Starting point is 01:55:52 consciousness and in complex systems and i don't think it goes beyond that i agree and uh one of the issues is that the cov does not explain how consciousness works in the first place. Another problem is that it has intrinsic problems that it's either going to violate the Church-Schubing thesis or it's going to be epiphenomenonist for purely logical reasons. It's a very technical argument against it. The fact that most philosophers don't seem to see this is not an argument in favor of philosophy right now at the level of which it's being done

Starting point is 01:56:25 and I approve of the notion of philosophy divesting itself from theories that don't actually try to explain but they pretend to explain and don't mathematically work out and try to compensate this by looking like a theory by using

Starting point is 01:56:41 Greek letter mathematics to look more impressive or to make pseudo predictions and so on, because people ask you to. But it's also not really Tononi's fault. I think that Tononi is genuinely seeing something that he struggles to express. And I think it's important to have him in the conversation. And I was a little bit disappointed by the letter, because it was not actually engaging with the

Starting point is 01:57:06 theory itself at a theoretical level that I would thought was adequate to refute it or to deal with it. And instead, it was much more like a number of signatures being collected from a number of people who later on instantly flipped on a dime when the pressure went another way. And this basically looked very bad to me that you get a few hundred big names in philosophy to sign this, only half of them later on coming out and saying, this is not what we actually meant. So I think that it shows that not just the IIT might be a pseudoscience, but there is something amiss in the way in which we conduct philosophy today. And I think it's also understandable because it is

Starting point is 01:57:48 a science that is sparsely populated, so we try to be very inclusive of it. It's similar to AGI in the old days. And at the same time, we struggle to discern what's good thinking and what's deep thinking versus these are people who are attracted to many of these questions

Starting point is 01:58:03 and are still trying to find the right way to express them in a productive way. I think that, I mean, phi as a measure is fine. It's not the be-all end-all. It doesn't do everything that's been attributed to it. And I guess anyone who's into the science of consciousness pretty much can see that already, that the frustrating thing is that average people who can't read an equation and don't know what's going on, being told, like, oh, the problem of consciousness is solved. And that can be a bit frustrating,

Starting point is 01:58:40 because when you look at the details, it's like, well, this is kind of interesting. But no, it doesn't quite quite do quite do all all that so but i mean why why people got hyped about that instead of much more egregious instances of bullshit is a is a cultural question which we we don't have time to go into now well thank you again thank Thank you both. Alright. Thanks a lot. Thank you. The podcast is now concluded. Thank you for watching. If you haven't subscribed or clicked that

Starting point is 01:59:11 like button, now would be a great time to do so, as each subscribe and like helps YouTube push this content to more people. You should also know that there's a remarkably active Discord and subreddit for Theories of Everything where people explicate toes, disagree respectfully about theories, and build, as a community, our own toes. Links to both are in the description. Also, I recently found out

Starting point is 01:59:35 that external links count plenty toward the algorithm, which means that when you share on Twitter, on Facebook, on Reddit, etc., it shows YouTube that people are talking about this outside of YouTube, which in turn greatly aids the distribution on Reddit, etc. It shows YouTube that people are talking about this outside of YouTube, which in turn greatly aids the distribution on YouTube as well. Last but not least, you should know that this podcast is on iTunes, it's on Spotify, it's on every one of the audio platforms. Just type in theories of everything and you'll find it. Often I gain from re-watching lectures and podcasts, and I read that in the comments, hey, Toe listeners also gain from replaying, so how about instead re-listening on those platforms? iTunes, Spotify, Google Podcasts, whichever podcast catcher you use.

Starting point is 02:00:14 If you'd like to support more conversations like this, then do consider visiting patreon.com slash kurtjaimungal and donating with whatever you like. Again, it's support from the sponsors and you that allow me to work on Toe full time. You get early access to ad-free audio episodes there as well. For instance, this episode was released a few days earlier. Every dollar helps far more than you think. Either way, your viewership is generosity enough. Thank you.

Theories of Everything with Curt Jaimungal - Joscha Bach Λ Ben Goertzel: Conscious Ai, LLMs, AGI

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.