Microsoft Research Podcast - AI Frontiers: Rethinking intelligence with Ashley Llorens and Ida Momennejad

Episode Date: March 28, 2024

Principal Researcher Ida Momennejad brings her expertise in cognitive neuroscience and computer science to this in-depth conversation about general intelligence and what the evolution of the brain across species can teach us about building AI.Learn more:AI and Microsoft Research | Focus AreaEvaluating Cognitive Maps and Planning in Large Language Models with CogEval | Publication, October 2023Imitating Human Behaviour with Diffusion Models | Publication, May 2023Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games | Publication, April 2023Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation | Publication, July 2021Predictive Representations in Hippocampal and Prefrontal Hierarchies | Publication, January 2022The successor representation in human reinforcement learning | Publication, September 2017Encoding of Prospective Tasks in the Human Prefrontal Cortex under Varying Task Loads | Publication, October 2013

. I'm Ashley Lorenz with Microsoft Research. In this podcast series, I share conversations with fellow researchers about the latest developments in AI models, the work we're doing to understand their capabilities and limitations, and ultimately, how innovations like

these can have the greatest benefit for humanity. Welcome to AI Frontiers. Today I'll speak with Ida Momenejan. Ida works at Microsoft Research in New York City, at the intersection of machine learning and human cognition and behavior. Her current work focuses on building and evaluating multi-agent AI architectures, drawing from her background in both computer science and cognitive neuroscience. Over the past decade, she has focused on studying how humans and AI agents build and use models

of their environment. Let's dive right in. We are undergoing a paradigm shift where AI models and systems are starting to exhibit characteristics that I and, of course, many others have described as more general intelligence. When I say general in this context, I think I mean systems with abilities like reasoning and problem solving that can be applied to many different tasks, even tasks they were not explicitly trained to perform.

Despite all of this, I think it's also important to admit that we, and by we here I mean humanity, are not very good at measuring general intelligence, especially in machines. So I'm excited to dig further into this topic with you today, especially given your background and insights into both human and machine intelligence. And so I just want to start here. For you, Ida, what is general intelligence? Thank you for asking that. We could look at general intelligence from the perspective of history of cognitive science and neuroscience.

And in doing so, I'd like to mention its discontents as well. There was a time where general intelligence was introduced as the idea of a kind of intelligence that was separate from what you knew or the knowledge that you had on a particular topic. It was this general capacity to acquire different types of knowledge and reason over different things. And this was at some point known as G, and it's still known as G. There have been many different kinds of critiques of this concept, because some people said that it's very much focused on the idea of logic and a particular kind of reasoning. Some people made cultural critiques of it. They said it's very Western-oriented. Others said it's

very individualistic. It doesn't consider collective or interpersonal intelligence or physical intelligence. There are many critiques of it, but at the core of it, there might be something useful and helpful. And I think the useful part is that there could be some general ability in humans, at least the way that Ji was intended initially, where they can learn many different things and reason over many different domains, and they can transfer ability to reason over a particular domain to another. And then in the AGI or artificial general intelligence notion of it, people took this idea of many different abilities or skills for cognitive and reasoning and logic problem solving at once. There have been different iterations of what this means in different times. In principle,

the concept in itself does not provide the criteria on its own. Different people at different times provide different criteria for what would be the artificial general intelligence notion. Some people say that they have achieved it. Some people say we are on the brink of achieving it. Some people say we will never achieve it. However, there is this idea, if you look at it from evolutionary and neuroscience and cognitive neuroscience lens, that in evolution, intelligence has evolved multiple times in a way that is adaptive to the environment. So there were organisms that needed to be adaptive to the environment where they were. That intelligence has evolved in multiple different species.

So there's not one solution to it. And it depends on the ecological niche that that particular species needed to adapt to and survive in. And it's very much related to the idea of being adaptive of certain kinds of different kinds of problem solving that are specific to that particular ecology. There is also this other idea that there is no free lunch and then no free lunch theorem, that you cannot have one particular machine learning solution that can solve everything. So the idea of general artificial intelligence in terms of an approach that can solve everything, and there is one end to end training that can be useful to solve every possible problem that it has never seen before,

seems a little bit untenable to me, at least at this point. What does seem tenable to me in terms of general intelligence is if we understand and study the same way that we can do it in nature, the foundational components of reasoning, of intelligence, of different particular types of intelligence, of different particular skills, whether it has to do with cultural accumulation of reasoning and intelligence skills, whether it has to do with logic, whether it has to do with cultural accumulation of written reasoning and intelligence skills, whether it has to do with logic, whether it has to do with planning. And then working on the particular types of artificial agents that are capable of putting these particular foundational building blocks together in order to solve problems they've never seen before.

A little bit like putting Lego pieces together. So to wrap it up, to sum up what I just said, the idea of general intelligence had a more limited meaning in cognitive science, referring to human ability to have multiple different types of skills for problem solving and reasoning. Later on, it was also, of course, criticized in terms of the specificity of it and ignoring different kinds of intelligence. In AI, this notion has been having many different kinds of meanings. If we just mean it's a kind of a toolbox of general kinds of intelligence for something that can be akin to an assistant to a human, that could make sense. But if we go too far and use it

in the kind of absolute notion of general intelligence as it has to encompass all kinds of intelligence possible, that might be untenable. And also, perhaps we shouldn't think about it in terms of a lump of one end-to-end system that can get all of it down. Perhaps we can think about it in terms of understanding the different components that we have also seen emerge in evolution in different species. Some of them are robust across many different species, some of them are more specific to some species with a specific ecological niche or specific problems to solve. But I think perhaps it could be more helpful to find those cognitive and other interpersonal, cultural, different notions of intelligence, break them down into their foundational building blocks,

and then see how a particular artificial intelligence agent can bring together different skills from this kind of a library of intelligence skills in order to solve problems it's never seen before there are two concepts um that jump out at me based on what you said one is artificial general intelligence and the other is human-like intelligence or human level intelligence and you've referenced the fact that you you know, oftentimes we equate the two, or at least it's not clear sometimes how the two relate to each other. Certainly, human intelligence has been an important inspiration for what we've done, a lot of what we've done in AI, and in many cases, a kind of evaluation target in terms of how we measure progress or performance. But I wonder

if we could just back up a minute. Artificial general intelligence and human-like human-level intelligence, how do these two concepts relate to you? Great question. I like that you asked to me because I think it would be different for different people. I've written about this, in fact. I think human-like intelligence or human-level intelligence would require performance that is similar to humans, at least behaviorally, not just in terms of what the agent gets right, but also in terms of the kinds of mistakes and biases that the agent might have. It should look like human intelligence. For instance, humans show primacy bias, recency bias, variety of biases. And this seems like it's unhelpful in a lot of situations, but in some situations it helps to come with fast and frugal

solutions on the go. It helps to summarize certain things or make inferences really fast that can help in human intelligence, for instance. There is analogical reasoning. There are different types of intelligence that humans do. and compare that to, for instance, let's say just a large language model like GPT-4, you will see whether they find similar things simple and similar things difficult or not. When they don't find similar things easy or difficult, I think that we should not say that this is human-like per se, unless we mean for a specific task. Perhaps on specific sets of tasks, an agent can have human-level or human-like intelligent behavior. However, if we look overall, as long as there are particular skills that are more or less difficult for one or the other, it might be not reasonable to compare them. That being said, there are many things that some AI agent and even a program language would be better at humans at. Does that mean that they are generally more intelligent? No, it doesn't, because there are also many things that humans do. We are very energy efficient. With very little amount

of energy consumption, we can solve very complicated problems. If you put some of us next to each other or at least give a pen and paper to one of us, this can be even a lot more effective. However, the amount of energy consumption that it takes in order for any machine to solve similar problems is a lot higher. So another difference between human-like intelligence or biologically inspired intelligence and the kind of intelligence that is in silico is efficiency, energy efficiency in general. And finally, the amount of data that goes into a current state of AI versus perhaps the amount of data that a human might need to learn new tasks or acquire new skills seem to be also different. So it seems like there are a number of different approaches to comparing human and machine intelligence and deriving what are the criteria for machine intelligence to be more human-like. But other than the conceptual aspect

of it, it's not clear that we necessarily want something that's entirely human-like. Perhaps we want in some tasks and in some particular use cases for the agent to be human-like, but not in everything. You mentioned some of the ways in which human intelligence is inferior or has weaknesses. You mentioned some of the weaknesses of human intelligence, like recency bias. What are some of the weaknesses of artificial intelligence, especially frontier systems today? You've recently published some works that have gotten into new paradigms for evaluation, and you've explored some of these weaknesses. And so can you tell us more about that work and about your view on this? Certainly. So, inspired by a very long-standing tradition of evaluating cognitive

capacities, those Lego pieces that bring together intelligence that I was mentioning, in humans and animals, I have conducted a number of experiments first in humans and built reinforcement learning models over the past more than decade on the idea of multi-step reasoning and planning. It is in the general domain of reasoning, planning, and decision making. And I particularly focused on what kind of memory representations allow brains and reinforcement learning models inspired by human brain and behavior to be able to predict the future and plan the future and reason over the past and the future seamlessly using the same representations. Inspired by the same research that goes back in tradition to Edward Tolman's idea of cognitive maps and latent learning in

the early 20th century, culminating in his very influential 1948 paper, Cognitive Maps in Rats and Men. I sat down with a couple of colleagues last year, exactly this time probably, and we worked on figuring out if we can devise similar experiments to that in order to test cognitive maps and planning and multi-step reasoning abilities in large language models. So I first turned some of the experiments that I had conducted in humans and some of the experiments that were done by Edward Tolman on the topic in rodents and turned them into prompts for chat GPT. That's where I started with GPT-4. The reason I did that was that I wanted to make sure that I will create some prompts that have not been in the training set. My experiments, although the papers have been published, the stimuli of the experiments were not linguistic. They were visual sequences that the human would see and they would have to have some reinforcement learning and learn about

from the sequences to make inference about relationships between different states and find what is the path that would give them optimal reward. Very simple human reinforcement learning paradigms, however, with different kind of structures. The inspirations that I had drawn from the cognitive maps works by Edward Tolman and others was in this idea that in order for a creature, whether it's a rodent, a human, or a machine, to be able to reason in multi-steps, plan, and have cognitive maps, which is simply a representation of the relational structure of the environment. In order for a creature to have these abilities or these capacities, it means that the creature needs to be sensitive and adaptive to local

changes in the environment. So I designed the sort of the initial prompts and recruited a number of very smart and generous with their time colleagues who we sat together and created these prompts in different domains. For instance, we also created social prompts. We also created the same kind of graph structures, but for reasoning over social structures. For instance, I say, Ashley is friends with Matt. Matt is friends with Michael. If I want to pass a message to Michael, what is the path that I can choose? Which would be, I have to tell Ashley, Ashley will tell Matt, Matt will tell Michael. This is very similar to another paradigm, which was more like a maze, which would be similar to saying, is a castle, it has 16 rooms, you enter room one, you open the door, it opens to room two, in room two you open the door, and so on and so forth.

So you describe using language the structure of a social environment or the structure of a spatial environment, and then you ask certain questions that have to do with getting from A to B in this social or spatial environment from the LLM. Or you say, oh, you know, Matt and Michael don't talk to each other anymore. So now in order to pass a message, what should I do? So I need to find a detour. Or for instance, I say, you know, Ashley has become close to Michael now. So now I have a shortcut. So I can directly give the message to Ashley and Ashley can directly give the message to Michael. My path to Michael is shorter now. So finding things like detours, shortcuts, or if the reward location changes,

these are the kinds of changes that, inspired by my own past work and inspired by the work of Tolman and others, we implemented in all of our experiments. This led to 15 different tasks for every single graph. And we had six graphs total of different complexity levels with different graph theoretic features. And each of them, we had three domains. We had a spatial domain that was with rooms that had orders, like room one, room two, room three, a spatial domain that there was no number, there was no ordinal order to the rooms, and a social environment where it was the names of different people, and so the reasoning was over social sort of spaces. So you can see this is a very large number of tasks. It's six times, 15 times, three, and each of the prompts we ran 30 times

for different temperatures, three temperatures, 0.5 and 1. And for those who are not familiar with this, a temperature of a large language model determines how random it will be or how much it will stick to the first or the best option that comes to it at the last layer. And so when there are some problems that maybe the first obvious answer that it finds are not good, perhaps increasing the temperature could help. Or perhaps a problem that needs precision, increasing the temperature would make it worse. So based on these ideas, we also tried it for different temperatures. And we tested eight different language models like this in order to systematically evaluate their ability for this multi-step reasoning and planning. And the

framework that we use, we call it CogEval. And CogEval is a framework that's not just for reasoning and multi-step planning, other tasks can be used in this framework in order to be tested as well. And the first step of it is always to operationalize the cognitive capacity in terms of many different tasks, like I just mentioned. And then the second task is designing the specific experiments with different domains, like spatial and social, with different structures, like the graphs that I told you, and with different kind of repetitions, and with different tasks, like the det that I told you, and with different kind of repetitions and with different tasks, like the detour shortcut, the reward revaluation, transition revaluation, and just traversal, the different tasks that I mentioned. And then the third step is to generate many prompts

and then test them with many repetitions using different temperatures. Why is that? I think something that Sam Altman had said is relevant here, which is sometimes with some problems, you ask GPT-4 100 times, and one out of those 100s, it would give the correct answer. Sometimes 30 out of 100, it will get the correct answer. You obviously wanted to give 100 out of 100 the correct answer, but we didn't want to rely on just one try and miss the opportunity to see whether it could give the answer if you probed it again. And in all of the eight large language models, we saw that

none of the large language models was robust to the graph structure, meaning its performance got really worse as soon as the graph structure didn't even have many nodes, but just had a tree structure that was six or seven nodes. Or a six or seven node tree was much more difficult for it to solve than a graph that had 15 nodes, but had a simpler structure that was just two lines. We noted that sometimes counter-intuitively, some graph structures that you think should be easy to solve were more difficult for them. On the other hand, they

were not robust to the task set. So the specific tasks that we tried, whether it was detour, shortcut, or it was reward revaluation or traversal, it mattered. For instance, shortcut and detour were very difficult for all of them. Another thing that we noticed was that all of them, including GPT-4, hallucinated paths that didn't exist. For instance, there was no door between room 12 and room 16. They would hallucinate that there is a door and they would give a response that includes that door. Another kind of failure mode that we observed was that they would fail to even find a one-step path. Let's say between room 7 and 8, there is a direct door. We would say, what is the path from 7 and 8?

And they would take a longer path to go from it. And the final mode that we observed was that they would sometimes fall in loops, even though we would directly ask them to find the shortest path. They would sometimes fall into a loop on the way to getting to their destination, which obviously you shouldn't do if you're trying to find the shortest path. That said, there are two different notions of accuracy here. You can have satisfying, which means you get there, you just take a longer path. And there is this notion that you cannot get there because you used some imaginary path or you did something that didn't make sense.

And you sort of gave a nonsensical response. We had both of those kinds of issues. So we had a lot of issues with giving nonsensical answers, repeating the question that we were asking, producing gibberish. So there were numerous kinds of challenges. What we did observe was that GPT-4 was far better than the other LLMs in this regard, at least at the time that we tested it. However, this is obviously on the basis of the particular kinds of tasks that we tried. In another study, we tried Tower of Hanoi, which is also a classic cognitive science approach to test of planning abilities and hierarchical planning abilities. And we found

that GPT-4 does between 0 and 10% in the three-disc problem and 0% for the four-disc problem. And that is when we started to think about having more brain-inspired solutions to improve that approach. But I'm going to leave that for next. So it sounds like a very extensive set of experiments across many different tasks and with many different leading AI models. And you've uncovered a lack of robustness across some of these different tasks. One curiosity that I have here is, how would you assess the relative difficulty of these particular tasks for human beings? Would all of these be relatively easy for a person to do, or not so much?

Great question. So I have conducted some of these experiments already and have published them before. Humans do not perform symmetrically on all these tasks, for sure. However, for instance, Tower of Hanoi is a problem that we know humans can solve. People might have seen this. It's three little rods that are unusually, it's a wooden structure. So you have a physical version of it, or you can have a virtual version of it. And there are different disks with different colors and sizes. There are some rules. You cannot put certain disks on top of others. So there is a particular

order in which you can stack the disks. Usually what happens is that all the disks on our one side, and when I say a three disk problem, it means you have three total discs. And there is usually a target solution that you are shown and you're told to get there in a particular number of moves or in a minimum number of moves without violating the rules. So in this case, the rules would be that you wouldn't put certain discs on top of others. And based on that that you're expected to solve the problem and the performance of GPT-4 on Tower of Hanoi three disc is between zero to ten percent and on Tower of Hanoi four discs is zero percent zero shot with the help it can get better with some some support it gets better so in this regard it seems like Tower of Hanoi is extremely difficult for gpt4

it doesn't seem as difficult as it is for gpt4 for humans it seems for some reason uh that it couldn't even improve itself when we explain the problem even further to it and explain to it what it did wrong sometimes uh if people want to try it out they should um sometimes it would argue back and say no you're wrong i did this right which was a very interesting moment for us with ChatGPT. That was the experience that we had for trying it out first without giving it sort of more support than that. But I can tell you what we did next, but I want to make sure that we cover your other questions. But just to wrap this part up, inspired by tasks that have been used for evaluation of cognitive capacities, such as multi-step reasoning and planning in humans, it is possible to evaluate cognitive

capacities and skills, such as multi-step reasoning and planning also in large language models. And I think that's the takeaway from this particular study and from this general cognitive science-inspired approach. And I would like to say also, it is not just human tasks that are useful. Tolman's tasks were done in rodents. A lot of people have done experiments in fruit flies, in. elegans, in worms, in various kinds of other species that are very relevant to testing as well. So I think there is a general possibility of testing particular intelligence skills, evaluating it inspired by experiments and evaluation methods for humans and other biological species. Let's explore the way forward for AI from your perspective.

As you've described your recent works, it's clear that your work is deeply informed by insights from cognitive science, insights from neuroscience. And recent works, your recent works have called for the development, for example, of a prefrontal cortex for AI. And I understand this to be the part of the brain that facilitates executive function. How does this relate to extending the capabilities of AI, a prefrontal cortex for AI? Thank you for that question. So let me start by reiterating something I said earlier, which is the brain didn't evolve in a lump. There were different components of brains and nervous systems and neurons that evolved at different evolutionary scales. There are some parts of the brain that appear in many different species, so they're robust

across many species. And there are some parts of the brain that appear in some species that had some particular needs, some particular problems they were facing or some ecological niche. What is, however, in common in many of them is that there seems to be some kind of a modular or multi-component aspect to what we call higher cognitive function or what we call executive function. And so the kinds of animals that we ascribe some form of executive function of sorts to seem to have brains that have parts or modules that do different things. It doesn't mean that they only do that. It's not a very extreme Fodorian view of modularity, but it is the view that, broadly speaking, when, for

instance, we observe patients that have damage to a particular part of their prefrontal cortex, it could be that they perform the same on an IQ test, but they have problems holding their relationship or their jobs. So there are different parts of the brain that selective damage to those areas because of accidents or coma or such, it seems to impair specific cognitive capacities. So this is what very much inspired me. I have been investigating the prefrontal cortex for, I guess, 17 years now, which is a scary number to say. But basically since I started my PhD and even during my master's thesis, I've been focused on the role of the prefrontal cortex in our ability for long-term reasoning and planning. And not just this moment, long-term open-ended reasoning and planning. And inspired by this work, I thought, okay, if I want to improve GPT-4's performance on,

let's say, Tower of Hanoi, can we get inspired by this kind of multiple roles that different parts of the brain play in executive function, specifically different parts of the neocortex and specifically different parts of the prefrontal cortex, part of the neocortex in humans? Can we get inspired by some of these main roles that I have studied before and ask GPT-4 to play the role of those different parts and solve different parts of the planning and reasoning problem, the multi-step planning and reasoning problem, using these roles and particular rules of how to iterate over them. For instance, there is a part of the brain called anterior cingulate cortex. Among other things, it seems to be involved in monitoring for errors and signaling when there is a need to

exercise more control or move from what people like to call a faster way of thinking to a slower way of thinking to solve a particular problem. So let's call the cognitive function of this part, let's call it the monitor. This is a part of the brain that monitors for when there is a need for exercising more control or changing something because there is an error maybe. There is another part of the brain and the frontal lobe that is, for instance, dorsolateral prefrontal cortex. That one is involved in working memory and coming up with simpler plans to execute. Then there is the ventromedial prefrontal cortex that is involved in the value of states and predicting what is the next state and integrating it with information from other parts of the brain to figure out what is the value.

So you put all of these things together, you can basically write different algorithms that have these different components talking to each other. And we have in that paper also written in a pseudocode style the different algorithms that are basically akin to a tree search, in fact. So there is a part of the multi-component or multi-agent realization of a prefrontal cortex-like GPT-4 solution. One part of it would propose a plan. The monitor would say, thanks for that. Let me pass it on to the part that is evaluating what is the outcome of this and what's the value of that and get back to you. It evaluates there and

comes back and says, you know, this is not a good plan. Give me another one. And in this iteration, sometimes it takes 10 iterations, sometimes it takes 20 iterations. This kind of council of different types of rules, they come up with a solution that is solving the Tower of Hanoi problem. And we managed to bring the performance from 0 to 10 in GPT-4 to, I think, about 70% in Tower of Hanoi three disks, and OOD, or out-of-distribution generalization, without giving any examples of a four-disk, it could generalize to above 20% in four-disk problems. Another impressive thing that happened here, and we tested it on the COG eval and the planning tasks from the other experiment too, was that it brought all of the sort of hallucinations from about 20 to 30 percent, in some cases much more, much higher percentages to zero percent. So we had slow thinking, we had

30 iterations, so it took a lot longer. This is, you know, fast and slow thinking, this is very slow thinking. However, we had no hallucinations anymore, And hallucination in Tower of Hanoi would be making a move that is impossible. For instance, putting a kind of a disc on top of another that you cannot do because you violate a rule or taking out a middle disc that you cannot pull out, actually. So those would be the kinds of hallucinations in Tower of Hanoi. All of those also went to zero. And so that is one thing that we have done already, which I have been very excited about. So you painted a pretty interesting, fascinating really picture of a multi-agent framework where different instances of an advanced model like GPT-4 would be prompted to play the roles of different parts of the brain and kind of work together.

And so my question is a pragmatic one. How do you prompt GPT-4 to play the role of a specific part of the human brain? What does that prompt look like? Great question. I can actually, well, we have all of that at the end of our paper, so I can even read some of them if that was of interest. But just a quick response to that is you can basically describe the function that you want the LLM, in this case GPT-4, to play, you can write that in simple language. You don't have to tell it that this is inspired by the brain. It is completely sufficient to just basically provide certain sets of rules in order for it, in order to be able to do that. For instance, after you provide the problem

sort of description, let me see if I can actually read some part of this for you. For instance, you give it a problem and you say, consider this problem. Rule one, you can only move a number if it's at this and that. You clarify rules here are examples here are proposed moves and then you say for instance your role is to find whether this particular number generated by as a solution is accurate in order to do that you can call on this other function which is the predictor and evaluator that sees, okay, if I do this, what state do I end up in and what is the value of that state? And you get that information. And then based on that information, you decide whether the proposed move for this problem is a good move or not. If it is, then you pass a message that says, all right, give me the next

step of the plan. If it's not, then you say, okay, this is not a good plan, propose another plan. And then the part that plays the role of, hey, here's the problem, here are the rules, propose the first step towards the sub-goal, or find the sub-goal towards this and propose the next step. And that one receives this feedback from the monitor, and monitor has asked the predictor and evaluator hey what happens if i do these things and what would be the value of that in order to say hey this is not a great idea so in a way this is a becomes a very simple prefrontal cortex inspired multi-agent system all of them are within the same gp sort of different calls to gpt4 but the same instance just like because we're calling it

in a code, it's just, you just call, it's called multiple times. And each time with this kind of a very simple in-context learning text that in text, it describes, hey, here's the kind of problem you're going to see. Here's the role I want you to play. And here's what other kind of roles you need to call in order to play your role here. And then it's up to the LLM to decide how many times it's going to call which components in order to solve the problem. We don't decide. We can only decide, hey, cap it at 10 times, for instance, or cap it at 30 iterations, and then see how it performs. So, Ida, what's next for you and your research? Thank you for that. I have always been interested in understanding minds and making minds. And this has been something that I've wanted to do since I was a teenager. And I think that my approaches in cognitive neuroscience have really helped me to understand minds to the extent that is possible.

And my understanding of how to make minds comes from basically the work that I've done in AI and computer science since my undergrad. What I would be interested in is, and I've learned over the years that you cannot think about the mind in general when you're trying to isolate some components and building them, is that my interest is very much in reasoning and multi-step planning, especially in complex problems and very long-term problems, and how they relate to memory, how the past and the future relate to one another. And so something that I would be very interested in is making more efficient types of multi-agent brain-inspired AI, but also to train smaller large language models, perhaps using the process of reasoning in order to improve their reasoning abilities. Because it's one thing to train on outcome, and outcome can be input and output, and that's the most of the training data that LLMs receive. But it's an

entirely different approach to teach the process and probe them on different parts of the process as opposed to just the input and output. So I wonder whether with that kind of an approach, which would require generating a lot of synthetic data that relates to different types of reasoning skills, whether it's possible to teach LLMs reasoning skills. And by reasoning skills, I mean very clearly operationalized, similar to the CagEval approach, operationalized, similar to the CAG-EVAL approach, operationalized, very well researched, specific cognitive constructs that have construct validity, and then operationalizing in terms of many tasks. And something that's important to me is a very important idea and

part of intelligence that maybe I didn't highlight enough in the first part is being able to transfer to tasks that they have never seen before and they can piece together different intelligence skills or reasoning skills in order to solve them. Another thing that I have done and I will continue to do is collective intelligence. So we talked about multi-agent systems that they are playing the roles of different parts inside one brain, but I've also done experiments with multiple humans and how different structures of human communication leads to better memory or problem solving. Humans, also we invent things,

we innovate things in cultural accumulation, which requires on a lot of, some people do something, I take that outcome, take another outcome, put them together, make something. Someone takes my approach and add something to it, makes something else. So this kind of cultural accumulation, we have done some work on that with deep reinforcement learning models that share their replay buffer as a way of sharing skill with each other. However, as humans become a lot more

accustomed to using LLMs and other generative AI, basically generative AI would start participating in this kind of cultural accumulation. So the notion of collective cognition, collective intelligence, and collective memory will now have to incorporate the idea of generative AI being a part of it. And so I'm also interested in different approaches to modeling that, understanding that, optimizing that, identifying in what ways it's better. We have found both in humans and in deep reinforcement learning agents, for instance, that particular structures of communication that are actually not the most energy-consuming one, it's not all-to-all communication, but particular partially connected structures are better for innovation than others.

And some other structures might be better for memory or collective memory converging with each other in what shape and in what frequency of communication in order to solve larger sort of cultural accumulation problems. Well, that's a compelling vision. I really look forward to seeing how far you and the team can take it. And thanks for a fascinating discussion. Thank you so much.