Orchestrate all the Things - Rebooting AI: The journey toward robust artificial intelligence. Deep learning, knowledge graphs, and the future of AI. Featuring scientist, best-selling author, and entrepreneur Gary Marcus

Episode Date: January 4, 2021

Gary Marcus is one of the more prominent, and controversial, figures in AI In this in-depth conversation, we cover everything from his background and early work in cognitive psychology as a way to understand the human mind, to his critique on Deep Learning, a holistic proposal for robust AI, the role of background knowledge, and knowledge graphs in particular, and the future of AI

Welcome to Orchestrate All The Things. I'm George Anadiotis and we'll be connecting the dots together. Stories about technology, data, AI and media and how they flow into each other, shaping our lives. Gary Marcus is one of the more prominent and controversial figures in artificial intelligence. In this in-depth conversation, we cover everything from his background and early work in cognitive psychology as a way to understand the human mind, to his critique on deep learning, a holistic proposal for robust AI, the role of background knowledge and knowledge graphs in particular, and the future of AI. I hope you will enjoy this.

If you like my work and orchestrate all the things, you can subscribe to my podcast, available on all major platforms, my self-published newsletter also syndicated on substack hackernan medium and dzone or follow the straight all the things on your social media of choice okay so uh gary thanks a lot for uh agreeing to uh to join to have this uh this q a session and just uh just in ways of introduction, I may as well start by grounding you a little bit and saying that, well, you know, on the one hand, for the people that are into AI, you probably count as one of the biggest names, you know with alongside people like Joshua Fangio or Jan de Koon and the deep learning camp so to speak on the other hand for people who are not in AI if somebody came and said well you

know here's this person who is considered to be one of the leading top minds in AI and his actual background is in humanities. He's a cognitive psychologist. Some of them may find that strange. Personally, I don't find that strange and I think we should actually have more of humanities in AI.

But I wonder if you would like to say a few words on what triggered you. I know that Steven Inger was one of your mentors, and I was wondering if you'd like to say a few words on how you got started in this field. Sure. I mean, I'm not sure I would say I'm a humanities person

in the sense of being in an English literature department or something like that, but I certainly come to AI from a perspective of trying to understand the human mind. So even before I did any AI, or before I did any psychology or things like that, I actually was a computer programmer as a kid and became interested in AI as a teenager.

But I quickly became dissatisfied with the state of the art. This is back in the 80s. It wasn't very good. And I realized that humans were a whole lot smarter than any of the software that I could write. So I actually skipped the last couple of years of high school based on a translator that I wrote that worked from Latin into English. That was kind of one of my first serious AI projects. But I realized that I could do a semester's worth of Latin by using a bunch of tricks, but it wasn't really very deep and there wasn't anything else out there that was deep. And this eventually led me to studying human language acquisition, human cognitive development,

including with Steve Finker, who was my mentor when I was a PhD student. And the work that I did with him was on how human beings acquire even simple parts of language, like the past tense of English. And I spent a lot of time comparing neural networks that were popular then with the things that human children did. And then the kinds of neural networks that I talked about then sort of disappeared. And then they reemerged in 2012. And when they reemerged, I realized that they had all the same problems that I had criticized in some of my technical work studying children and studying adults and studying human beings. And I've spent a lot of the last decade

trying to look at what we know about how human children, how did they learn about the world and language and so forth, and what can that tell us about what we might need how human children, how do they learn about the world and language and so forth? And what can that tell us about what we might need to do in AI? Okay. That makes perfect sense. And to kind of follow up on that, you are one of those people that wear many, many hats. So besides the fact that you are a professor in NYU,

you're also an entrepreneur and a few other things. So I'll play an end and end. Obviously, you have a book that has just come out, if I'm not mistaken. So one of the things I was wondering about is how do these activities inform each other? So obviously you started out as a researcher. What brought on your evolution or your wandering in those other areas, so to speak? Well, I think even as a scientist, which I would say is the first thing that I am, I wear a lot of hats. So as a scientist, I tried to use the techniques of psychology, the knowledge of linguistics, the tools of computer science, the analytical thinking of

philosophy, and probably leaving something out, and the understanding of molecular biology to put together a synthesis. So I'm always moving from one hat to another and trying to bring different things together. So even before I did any of the other stuff, I was really trying to be an interdisciplinary cognitive scientist trying to bring together what we know from many fields in order to answer a really hard question, which is, how does the mind work? How does it develop? How did it evolve? In time, that led me to be a writer as well, because I found that people didn't speak each other's languages in these different fields. The only way I was going to bring these fields together was to get people to understand one another. And so that really pushed me hard to

become a writer, which I was not originally talented at, but I got good at over time and started writing for the New Yorker and so forth. And then while I was doing that writing and AI came back in fashion, I felt like everybody was doing things all wrong. And that actually led me to be an entrepreneur because I wanted to take a different approach on machine learning. And so I founded my first company in 2014. It was called Geometric Intelligence. We did some machine learning. And so I founded my first company in 2014. It was called Geometric Intelligence. We did some machine learning. Eventually, Uber acquired us relatively early in the history of the company, about two years after it was acquired. And there I helped to launch Uber AI Labs. Now I'm building a new company called Robust AI. Excuse me this new company, we're trying to build a smarter generation of robots

that can be trusted to act on their own rather than just being teleoperated. And rather than just working in assembly lines, they could work in a wide range of environments, whether that's a home or retail situation or elder care or construction and so forth. Okay. I just have one additional question. Oh, sorry. Can I say one more thing? Of course. Please do. Even now, you could say I'm on my third career, I suppose, as an entrepreneur. I'm constantly going back and forth between these things. So as an entrepreneur, I need to understand the science of AI, and we need to make good guesses about what might work.

A lot of what scientists do is really take partial information and make best guesses. And I have to do that all the time as an entrepreneur, including in the fields that I know well as a scientist about natural intelligence, artificial intelligence, and so forth. Yeah, that makes perfect sense and actually reminds me of something someone who's not really famous, just a good friend once said, that that's actually a very good way to explain how AI works to entrepreneurs because this is what they do every day. They make estimates based on past experience and knowledge. They just don't realize that they're doing it. Yeah, a good entrepreneur has to constantly update their so-called priors, their understanding of the world based on new data.

You have to do that every day as an entrepreneur, especially in an early stage startup. You're constantly taking partial information, making your best theory about what's going on, and trying to take the action that you think is going to most likely work under those circumstances. And as scientists, it's the same thing. So people talk about evolution as a theory. Well, it is a theory. It's a really good theory. It fits all the available data. But as a scientist, if suddenly there was some other piece of data that emerged, you would look at that piece of data. You know, scientists are ultimately agnostic about everything, always trying to integrate the information to make the best guess that you can make at that moment. Yeah, that's very true, actually. Another

interesting thing that you touched upon in your introduction was the re-emergence of neural networks. And as people who have a little bit of interest and background in the evolution of AI itself may know, this has been a kind of pendulum, if you want to call it that. So going all the way from the symbolic roots of AI to the gradual emergence and then reemergence of machine learning in its various form, whether it's deep learning or other approaches. And a very interesting aspect of that, these two camps, if you want to call them that, to me at least

has been your debate with a number of people from the deep learning camp. And what I find interesting in that is that, you know, a lot of it actually, in my humble opinion, has to do with very, very human aspects, like, you know, the fact that some people may have worked in obscurity for a long time and are now in the spotlight and vice versa. So I was wondering what your take is on that and how you personally feel about your role and how you see that having evolved and whether you think this is a natural thing, a good thing, and where are the boundaries

basically? Well, I guess there are different questions there. I mean, one is about what is the right science and engineering practice to bring us forward and another is sort of about the culture of science. And I guess you're asking more about that, at least at first. I would say that I find the culture of AI right now to be unfortunate. So you do have this set of people that were toiling in obscurity. They are now in power. And I feel like rather than learning from what it's like to be disregarded, they're disregarding a certain kind of courage. They're not the kind of courage that our healthcare workers have. I have gotten out onto a different kind of frontline and said, this is what we need to do.

It's not popular right now, but this is why the stuff that's popular isn't working. And that's led a lot of people to be irritated with me. But I think, you know, I learned from my father to stand up for what I believe in and that's what I do. And I think for a long time, that wasn't good for me personally, because I don't think a lot of people were listening to the side of the argument that I was representing. But I think over the last few years, there's been a sea change in the field.

People have realized that the limitations in deep learning that I've been pointing out since 2012 actually are real, that the hype that I was pointing to is real, and there's been too much of it. Now, even leaders of the deep learning field are acknowledging the hype and also acknowledging some of the technical limitations about generalization and extrapolation that I've been pointing out for a number of years. And so in the short term, from 2015 to 2018, I was subject to a lot of abuse. But I think that in the last year especially, a lot of people have come around to a view that's something like mine. So was that fun being the target of abuse?

No, but I think that AI has a lot to offer or could have a lot to offer if we did it better. And I'm pleased to see people starting to look at a broader range of ideas. And hopefully that will lead us to a better place. Yeah, it certainly isn't fun being in the epicenter for all the wrong reasons. Well, one of the leading AI research institutes, I won't name names, but they call me the antichrist there. You know, I mean, it's sort of flattering in a way to, you know, have such a reputation, but it's also not really productive to, you know, what's the word I'm looking for?

Sort of straw man a person rather than looking at the arguments. Yeah, yeah, indeed. Indeed. I mean, that's what word I'm looking for? Sort of straw man, a person rather than looking at the arguments. Yeah, yeah, indeed, indeed. I mean, that's what I was going to say, actually, that, you know, it's good to have attention, but that kind of attention can really, you know, be distracting. I mean, it's not all bad. I don't want to, it's certainly not all good. I wish we could get to a different place. But on the other hand, because there have been these debates that I've been in with Benjio and Lacoon and so forth, it's brought attention to the issues. So, you know, I went to NeurOps, which is the big neural networks conference in December. And people from all over the world knew exactly who I was and knew about these debates and

so forth. So it hasn't always been fun to be the target of knew about these debates and so forth. So it hasn't always been fun to be the target of hostility on Twitter and so forth. But I do think that there's been some value too, because the issues are much more prominent in a lot of people's minds. So I don't really go to conferences anymore and have people like, you know, not know who I am. They know who I am and they know what the arguments are about. And that's great. Okay. So let's actually talk about the arguments for the ones that do not know. And let's try to popularize them, if you will. So a lot of that could, I guess, be anchored around language models. And this has been a really popular event for popular science

kind of topic lately. And models such as BERT and GPT-2, which has even been labeled too dangerous to release by its makers, and the latest one, MINA, by Google, they have also drawn the attention of the general public. And in your work, you point out deficiencies in those models. My simplification of your argumentation would be that those deficiencies basically come

from the fact that those models, as elaborate as they may be, they're basically an approximation of language understanding. They emulate it, but they don't really possess it, so to speak. So GPT-2 is the model that The Economist recently interviewed as AI talks about the future. And I have a recent article called The Next Decade in AI, which is an archive. And maybe you could link that for your listeners or readers. The argument I make there is even a little bit sharper than what you just said. So these things are approximations, but what they're approximations to is the language use rather than language understanding. So you can get statistics about

how people have used language and you can do some amazing things if you have a big enough database, and that's what they've gone and done. So you can, for example, predict what's the next word likely to be in a sentence based on what words have happened in similar sentences over some very large database, gigabytes of data. And often locally, it's very good. So you say, what's the name of your favorite band? And Mina says, Avenge Sevenfold. And that turns out to be the name of an actual band. And so it kind of fits in the right context. The systems are very good at predicting categories. So if you say, I have three of these and two of these,

how many do I have at all? It will definitely give you a number. It will know that a number is supposed to come next because people use numbers in particular contexts. What these systems don't have at all is any real understanding about what they're talking about. And so I showed this in a benchmark that I started working on in December and that I've written about recently. Very simple questions show the difference between understanding kind of the general category of something that's going to

happen next and the details of what really happens in the world. So if I say, I put three trophies, excuse me, if I put three trophies on a table, and then I put another one on the table, how many do I have? Well, you as a human being can easily add up three plus one is four, you build a mental model in your head of how many things are there, and you can interpret that model. But these systems, if you say I put three trophies on the table and another one, how many are there might say seven or twelve like they know it's supposed to be a number but they don't actually understand that you're talking about a set of objects that are countable in a particular place i don't know how to do the counting in the that's one example that i had in the benchmark and then mina came out with an even bigger database and it had examples

like what's the name of your favorite band avenge Sevenfold then you ask the same system what's the name of your least favorite band and it again says Avenged Sevenfold any human would realize your favorite band and your least favorite band can't be the same thing unless you're lying or trying to be funny and these systems don't understand that so they understand kind of what category goes in a particular place because of language usage, but they don't understand what it is that they're talking about. And that's a really deep deficiency. And it's actually an old deficiency. It goes back to 1965 to the system ELISA, which just matched keywords and talked to people about therapy, basically. So it's kind of actually

depressing if you think that we're 55 years later and we still have basically the same problem. You know, people talk about, like Kurzweil talks about exponential progress, and there's been exponential progress in narrow AI, things like playing chess. But in things like language understanding, there's really not been that much progress. We still don't know how to make a general purpose system that could understand our conversation or a television show or a podcast or news article or anything like that in anything like a systematic way. You know, the counter argument to that is that, okay, you know, we just need more and more

data or more compute and eventually the system... It's a terrible counter argument it's common but that's why i mentioned mina so right um first of all gpt2 has like a thousand times more data or ten thousand times more data than you might have had like a year or two earlier and it produces more fluent speech but there's no more understanding about what's going on than the earlier systems and And then Mina is, I think, 10 or 20 times as much data as GPT-2, and it still doesn't understand what's going on. So if you plotted on some independent measure of comprehension, nobody's got a great one, but if they did, you could plot comprehension versus amount of data. Eliza had zero data. It was just matching keywords that are handwritten. GPT-2 has an immense amount of data, and MENA has, you know, 100 times that. And you would see that for real comprehension,

there's really been no progress at all. So there's much better, there's progress towards approximating the statistics of language use, although we had something called bigrams back in the day. But there's been progress on that. Bigger data sets naturally give you better approximations, but those approximations are yielding no progress whatsoever on the problem of language understanding. So I think a few people will agree, or at least partially agree with your critique.

But then again, the critique on your critique up to the main critique on your critique actually has been like, okay, so maybe some of these things actually are true, but it's not a constructive critique because it doesn't actually come out and suggest something some way out of this deadlock. But I think your latest paper actually does that.

So if you'd like to expand, actually it mentioned some specific steps. So if you'd like to expand. So this is, again, that article called The Next Decade, which came out in February. And, you know, unfortunately the world has much larger problems to deal with right now. So I don't think it's been as discussed as it might have been. But what I argued there is that we need four things that

really no current AI system combines. So there are little pieces and hints of all of this already, but nobody's putting them all together right now. So in the main tradition that's popular right now, deep learning, it's just you pour in a lot of data. And what's missing are four things. One is a connection to the world of classical AI. So I don't think we want to get rid of deep learning, but we want to use it in conjunction with some of the tools of classical AI. What classical AI is good at is representing abstract knowledge, representing sentences, or representing abstractions like stairs are things that you climb. So we know a lot of abstract things about the world that we can use in very general ways. So once you know that about stairs, you understand that it works

for wooden stairs and stone stairs and metal stairs and so forth. And so you don't just learn something about one particular staircase, you learn something general. So we need to have hybrid systems that can use the kind of perceptual information. This is what this looks like, and this is what it's called, that has been driving the success in deep learning with some of the kinds of knowledge structures that classical AI used in the 1950s through the 1980s that is kind of out of fashion right now but still needs to be there. And in fact, there are some systems that already do this, like Google search actually does use some of both, even though the fashionable thing right now is to just use the deep learning side of things. So we need hybrid models, that's number one. Number two is we need to have rich ways of specifying knowledge.

We need to have large-scale knowledge. So the average human being, well, take the six-year-old who just walked into my room as we're recording this. She's carrying a container containing pencils, like a cup with many pencils. So she knows that the cup holds the pencils. She knows if she turned over the cup that the pencils would all fall out. She knows that the pencils can be used for coloring. She knows that they need to be sharpened in order to do that. She knows a million things about just that one little tiny piece of the world that she just acted upon a moment ago.

She knows that she needs to be quiet while I'm recording the call, so she closed the door. Her world is filled with lots of little pieces of knowledge. Deep learning systems mostly aren't. They're mostly just filled with correlations between particular things. So they might be able to recognize a pencil, but they don't know what a pencil is for. They don't know what a cup is for. They don't know that even if you don't usually put pencils in cups that you could and what that would be a value for. So we need a lot of knowledge. Then we need to be able to reason about these things.

So we need to realize, for example, that if I cut a hole in the bottom of the cup, the pencils might well fall out. And human beings do this kind of reasoning all the time. Current AI systems just don't do it. And the last thing that we need are cognitive models, which are things inside our brain or inside of computers that tell us about the relations between the entities that we see around us in the world. So if we go back to the case I talked about before, they put three trophies on the

table and then two more. I have in my head some representation of this is how many things I might see on the table. It's like whenever you build a novel, you build something like a mental image of that. We need to have AI systems that do that routinely. In the Next Decade article, I talk about some systems that can do this some of the time and why the inferences that they can make about how the world works are far more sophisticated than what deep learning alone is doing. But I also point out that those existing systems are very much hand-wired to particular situations. They don't really work in a freely accessible way in the real world.

So there's quite a lot of work that still needs to be done. And actually, I know that there's been some pushback towards these notions that you're putting forward. And more specifically, again, I will refer to the recent debate where the counterargument to what you're saying is basically like, no, we don't really want to do that. We just want to encode up to four lines of code because, you know, this is what we do. This is what's acceptable. Can you find, you know, some reasonable explanation for that? Like, you know,

some pre-encoded knowledge is fine, but not above a certain threshold? I mean, I thought, so, I mean, you're referring to a specific thing that Bengio said that I think is unreasonable. I'll first say that a lot of what Bengio has said recently is very reasonable. So Joshua Bengio is one of the three founders of deep learning. I found his early work on deep learning to be kind of a little bit more on the hype side of the spectrum. I think he took the view that if we had enough data, it would solve all the problems. And he now sees that that's not true. And in fact, he's softened his rhetoric quite a bit. He's acknowledged that there was too much hype in the BBC interview in January. In our debate and the talk he gave just before, he acknowledged limits of generalization that I've

been pointing out for a long time, although he didn't attribute them to me, but it's another story. So he's recognized some of the limits. However, on this one point, I think he and I are still pretty different. We were talking about which things you need to build in innately into a system. So there's going to be a lot of knowledge. Not all of it's going to be innate. A lot of it's going to be learned,

but there might be some core that is innate. And he was willing to acknowledge one particular thing because he said, well, that's only four lines of computer code. He didn't quite draw a line and say nothing more than five lines, but he's like, you know, it's hard to encode all of this stuff. I think that's silly. We have gigabytes of memory now cost nothing, so you could easily accommodate the physical storage.

It's really a matter of building and debugging and getting the right amount of code. What I would say is that the genome is a kind of code that's evolved over a billion years to build brains autonomously without a blueprint. It's a very sophisticated system I wrote about in a book called The Birth of the Mind. And there's plenty of room in that genome to have some basic knowledge of the world. And that's obvious if you ever see what we call a precocial animal, like a horse, just get up and start walking, or an ibex that climbs down the side of the mountain when it's a few hours old. There has to be some innate knowledge there about what the visual world looks like, how to interpret

it, how forces apply to your own limbs and how that relates to balance and so forth. So there's a lot more than four lines of code in the human genome. In fact, most of our genome is expressed in our brain as the brain develops. So a lot of our DNA is actually about building strong starting points in our brains that allow us to then accumulate more knowledge. It's not nature versus nurture, like the more nature you have, the less nurture you have. And it's not like there's one winner there. It's actually nature and nurture work together. The more that you have built in, the easier it is to learn about the world. Yeah. I mean, obviously I don't think at this point in time in history,

we actually lack the means to record this knowledge. I would probably argue that the hard part is not so much, you know, how much storage we have for this knowledge, but how do we actually encode it and how do we get it there? And to tie that to a kind of favorite topic in mind and to be honest with you what was the main attraction that made me notice initially your more recent paper knowledge graph so to give a little bit of overview for people who may not be that familiar with that it's basically knowledge graphs are basically a rebranding of a rather old by now notion,

so the semantic web. It's again one of the ways of encoding knowledge, so knowledge representation and reasoning. The semantic web also was about reasoning and much like symbolic AI, it went through a trough of disillusionment, so to speak, and now it's kind of fancy again. So I wanted to get your opinion on, you know, whether this kind of technology can have a

role in this kind of hybrid approach that you're advocating. Yeah, I mean, one way to think about it is there's actually an enormous amount of knowledge that's represented on the internet that's available for free, essentially, that is not being leveraged by current AI systems. And much of that knowledge is problematic. And most of the world's knowledge is imperfect in some way or another. But there's an enormous amount of knowledge that, say, a bright 10-year-old can just pick up for free. And we should have our AIs be able to do that. And some examples are, first of all, Wikipedia says so much about how the world works. And

if you have the kind of brain that a human does, you can read it and learn a lot from it. If you're a deep learning system, you can't get anything out of that at all or hardly anything from it. And sort of Wikipedia is the stuff that's on the kind of front of the house. And the back of the house is things like semantic web that label web pages for other machines to use. There's all kinds of knowledge there, too. It's also being left on the floor by the current approaches. Obviously, the kinds of computers that we are dreaming of that can help us to, for example, put together medical literature or develop new

technologies are going to have to be able to read that stuff. We're going to have to get to AI systems that can use the collective human knowledge that's expressed in language form and not just as a spreadsheet in order to really advance, in order to make the most sophisticated systems. I think in terms of the specific project that's called the Semantic Web, turned out to be harder than we anticipated to get people to play along and be consistent about it and so forth, but that doesn't mean there's no value

in doing those kinds of things and having that kind of knowledge explicit and so forth. We just need better tools to make use of it. Yeah, I personally agree to that. And I would also add that this seems, you know, to have been realized by many people these days. So you see these actually mixed approaches. So yeah, it's become evident that you can't really expect people

to manually annotate each and every piece of content that they publish. So a lot of that is now happening kind of automatically or semi-automatically by content management systems and so on. For now, probably the best that we can do is to semi-automate these kinds of things. We don't really have AI that's as sophisticated as a person, and we can't have people sitting there annotating every single thing. And so, you know, there's some level of compromise. Over time, the machine annotation of these things will get better as the machines are more sophisticated and there'll be kind of an upward ratcheting effect as we get to AI that is more and more sophisticated. Right now, the AI is so unsophisticated, it's not really helping that much. But that will change over time. And I think more generally, you're right, people are

recognizing the value of hybrids, especially in the last year or two in a way that they just were not. People fell in love with this notion of, I just pour in all of the data and this one magic algorithm is going to get me there. And they thought I was going to solve driverless cars and chatbots and so forth. And there's been a wake up, some of it led by me and some of it independent, that, hey, that's not really working, that we need other techniques. And so I think there's been much more hunger to try different things and try to find the best of both worlds in the last couple of years, as opposed to maybe the five years before that. I would also point that, you know, there's actually also people going,

going to the other direction, basically. So you refer to Wikipedia earlier. As you may know, there's also the, the backend edition, let's say of Wikipedia. So basically what some people have done is they have built smart parsers that take advantage of the structural relations in Wikipedia and basically create a structured version of that, which is called DBpedia. And this is actually a very, very useful resource that many people use for machine learning.

They actually feed their models with DBpedia encoded knowledge and they make them better that way. So I wonder if you have an opinion on that. Well, a couple of things. I mean, one is that the stuff that we use right now the most is the most structured part. So like the things in those boxes in Wikipedia that are very structured, those are the things that are most accessible to current techniques. They're already somewhat useful for things like disambiguating what a particular use of a word is going to be.

There's a lot of knowledge in Wikipedia that's in the form of unstructured text that doesn't go in those boxes. And we're not nearly as good as leveraging that kind of stuff. So if you have a historical description of what somebody did during some particular war, the system's probably not going to be able to understand that at this moment. But it will be able to look up that this person's title was Captain. They were alive during these years. They died in this year.

The names of their children were this and that. So the latter stuff that's more structured, as I'm sure you know, is more easily leveraged by the current techniques. And there's a whole lot of other stuff that we're not using. Well, the latter stuff, I'm glad to see that we're starting to use some of it. Even there, I don't think we're using it as well as one could in principle. Because if you don't understand the conceptual relations between all these entities, it's to maximize the use that you you get out of it you're right in saying that much of the unstructured content is not really leveraged that much as the structured one one interesting approach that i've seen emerging over the last couple of years is embeddings basically graph embeddings and even more specifically,

RDF embeddings. I wonder if you're aware of those and what you think of those. Embeddings where you turn these things into neural networks, now embedding, or you have something different in mind? No, basically using vectors and vector distances. Right, so you're mapping these things onto vectors.

There's the famous quote, I won't get it exactly verbatim, from Ray Mooney, which is, you can't stuff the entire meaning of a sentence into a fucking vector. I'm sympathetic to it. Vectors, at least as we understand them right now, often take a lot of different things, make a similarity measure around that, but don't really represent things with precision.

And so they're often a mixed bag. You get something out of them, but you don't know exactly why. I mean, the most famous example I would say was when people did all this word-to-vec mathematics a couple of years ago. So they were like, wow, I have this system. It makes vectors based on all these words.

And I teach it king is to queen, and then it infers, I forget what the example is, but I'll just make it up, princess is to princess. And sometimes that stuff works, but it's not really all that reliable. I've yet to see that kind of architecture be supremely reliable. It often gives you some vague notion of similarity,

but often without the precision that a person would have if they were thinking about the same set of concepts. Okay, sorry now. Okay, so since I think we only have a few minutes left, I'm going to wrap up by giving you a few short ones. And let's start with AI chips, basically. So seeing the issue of AI and whether it should be modeled

after the human brain in your paper, I like the way you approach it. So basically, the reason I approach it so basically I would the reason I liked it was because it's sincere what to me what it seemed you're saying is like look we don't really know how our brain works but whatever that way may be it doesn't necessarily mean that we're going to build we should build AI to to mimic our brain and this made me think of neuromorphic chips, which some people are working on building. And, you know, generalizing this made me think of AI chips in general. There's

been lots of progress there and lots of progress in, you know, in AI models is actually being driven. It's a kind of, you know, feedback loop with progress in chips, feeding AI models and vice versa actually because some people are trying to build specific chips to perform better for the models and I wonder if you have a take on those. Alright, well first let me

clarify what's obvious to you but maybe not to your audience. So my view is not that we should be imitating human brains but that we should be imitating human brains, but that we should be learning from them or human minds. We should be learning from them, but not replicating them. So the best AI systems will have some of the properties of human minds and some properties of machines. They will put them together in new ways that exceed either what we could do

with current machines or with current human brains. In the case of neuromorphic chips, the idea is to learn from how the brain works in order to make better chips. So far, I'm totally sympathetic in principle. I think the reality is we don't know enough about neuroscience yet to make that work all that well. And I worry about people like Jeff Hawkins who try to stick only to the things we already know about the brain, because I think we just don't know enough about the brain to really do that effectively yet. Maybe 20 years from now, we will be able to do that. But right now, our understanding of brain operation is pretty limited. And as a consequence, I think that the neuromorphic field has been more promised than results. So there's not a lot of concrete applications from it yet.

We have some reason to think that it might lead us, for example, to lower power alternatives to the technologies that we're using right now. And that may come to pass. So far, I haven't seen anything really that useful come out of that literature. It will, but maybe we need to know a little bit more about how the brain works before we can really leverage that. Okay. Another, something else that your approach and things made me think about was what people

called software 2.0. So to briefly explain the traditional approach to software has basically been to build algorithms that encode in really, really detailed way, you know, what should happen and in what sequence. And software 2.0 kind of says well okay for really complex processes it's really hard or even impossible to do that so let's just throw lots of data to some machine learning platform which is going to figure out the pattern and it's going to produce something that we can use in the end and this kind of this brings many issues to

the traditional software development process because you know people are used to testing and developing software in the traditional way and to change the paradigm so drastically kind of makes people leave people scratching their heads as to how exactly they should test those systems and kind of touches on explainability, I would say. So I wonder what you think about this approach and its challenges, and if you have an opinion on how the software industry

may or may not overcome that. I don't think that that can work as a general technique. So for example, nobody tries to build a web browser by taking supervised learning over a bunch of logs of what users typed and what they saw on their screens. Like in principle, that's what machine learning approach would be,

is rather than sit there and laboriously coding all of the conditions under which different plugins work and blah, blah, blah, you would just induce it from the data. And that doesn't really work. Nobody's even trying to make that work. They're doing stuff where you basically do things on machine learning and large data sets

for narrow cases like categorization. So you do it for speech recognition, which syllable is that? And in those cases, it actually works better than classical techniques. But if you want to build an application like a web browser or word processor or an operating system, you still need a lot of the classical techniques. So it's nice that we have some new techniques available on the side, but people who think we're just going to, like, we're not going to need anybody to code, well, certainly in the short term, that's just not true. I think that the real revolution might come, but it's going to be a

long time, from what Charles Simone called intentional programming, which is instead of writing all the lines of code that you want, you have the machine figure out what is the logic of what you want to do. And maybe you do that with some machine learning and some classical logically driven programming, but we're not anywhere close to being able to do that well. Okay. And to wrap things up, kind of a forward-looking question. So, what are you basically trying to achieve with robust AI, if you can summarize that? And basically my main question would be why choose robotics? And to kind of give you a prompt on that, I recently saw Jan LeCun's

statement on why Facebook is actually also pursuing some efforts in this area. And his take was that it's not so much about the robots themselves, but more about the challenge of having to deal with multi-sensory input in real time. So, yeah. I often see things somewhat similarly, but not entirely similarly to Jan. I think robots are really, really interesting because they force you beyond approximation towards systems that really can cope with the real world. So if you're dealing with speech recognition, you can solve the problem by just gathering a lot of data because words don't change that much from one day to the next.

But if you want to build a robot that can, say, wander the streets and clean them up, it has to be able to deal with the fact that every street's going to be different every hour, every day. Robots have to be as resourceful as people if we're going to put them out in the real world. Right now they're mostly on things like assembly lines, very well-controlled environments with either no humans or humans are limited to a particular place and so forth. So they're very

restricted in what they can do right now. And that allows us to sidestep the question of how do you make robots that are really autonomous and able, which is part of the definition of being a robot, and able to deal with things on their own. And I think that's a fascinating intellectual problem. And I think it's one that will push the field of AI forward considerably as we move robots more and more into the real world. As a function of a business, I think it's a huge opportunity because there's not that much of the world that actually is automated with robots right now and eventually be huge.

Every single person, for example, may eventually have their own domestic robot just like they have a cell phone because these things could be incredibly useful if we could build them in a way that is safe and reliable. So I think there's, you know, robotics right now is maybe a $50 billion a year industry. It could be much, much bigger.

But in order to get to that place we need to make them safe and reliable and trustworthy um make them flexible and that's what my company is trying to do is is to build a platform that allows us to build robots that are much more flexible and robust in the real world. Okay, one last follow-up on that. So for the last company you were building, you got an acquisition offer and you took it. How about this one? Somebody came along tomorrow morning and offered you a good sum

and the kind of environment, you know, analogous to what you were offered by Uber in your previous company, would you take it? I mean, I guess, you know, as a CEO, you have to take any offer seriously. But we have pretty clear ideas about what we want to build and, you know, we'd like to be able to build it. If somebody gave us a lot of resources and, you know, then I guess we would consider it. But our objective here is not to get acquired. Our objective is to reshape the field and we'll see what the best way is to get there.

Okay. Well, great. Thank you very much. It's been a great pleasure in the meantime. Actually, I had a very bad day and this was the best part of my day. Thanks a lot. Glad to have cheered you up. It was fun for

me. Take it easy. If you need clarification by email, let me

know. Thank you very much. Enjoy the rest of your day. Thank you. Bye-bye.

Bye-bye. Thanks for sticking around. For more stories like this, check the link in bio and

follow link data orchestration.